2020-11-19T03:06:25.118 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-19T03:06:25.120 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --verbose --block --owner scheduled_yuvalif@teuthology --archive /home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633886 --name yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi --lock -- /tmp/teuthology-worker.o9q5gh0k.tmp 2020-11-19T03:06:25.160 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T03:06:25.236 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633886 branch: wip-yuval-fix-47904 description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: null first_in_suite: false job_id: '5633886' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-yuval-fix-47904 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-yuval-fix-47904 sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 owner: scheduled_yuvalif@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuval-fix-47904 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.1594 2020-11-19T03:06:25.237 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa; will attempt to use it 2020-11-19T03:06:25.238 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa/tasks 2020-11-19T03:06:25.239 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-19T03:06:25.241 INFO:teuthology.task.internal:Checking packages... 2020-11-19T03:06:25.272 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '409eac507e130adcc9fee71aa5fd16ed7f289313' 2020-11-19T03:06:25.273 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-19T03:06:25.274 INFO:teuthology.packaging:ref: None 2020-11-19T03:06:25.274 INFO:teuthology.packaging:tag: None 2020-11-19T03:06:25.275 INFO:teuthology.packaging:branch: wip-yuval-fix-47904 2020-11-19T03:06:25.276 INFO:teuthology.packaging:sha1: 409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T03:06:25.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-yuval-fix-47904 2020-11-19T03:06:25.451 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7199-g409eac50-1bionic 2020-11-19T03:06:25.451 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-19T03:06:25.476 INFO:teuthology.task.internal:no buildpackages task found 2020-11-19T03:06:25.477 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-19T03:06:25.628 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-19T03:06:25.630 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T03:06:25.707 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:06:35.727 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:06:45.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:06:55.792 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:07:05.808 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:07:15.829 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:07:25.851 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:07:35.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:07:45.902 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:07:55.931 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:08:05.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:08:15.965 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:08:25.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:08:36.003 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:08:46.013 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:08:56.038 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:09:06.058 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:09:16.074 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:09:26.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:09:36.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:09:46.134 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:09:56.162 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:10:06.196 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:10:16.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:10:26.245 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:10:36.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:10:46.298 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:10:56.316 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:11:06.336 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:11:16.357 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:11:26.376 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:11:36.401 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:11:46.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:11:56.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:12:06.459 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:12:16.477 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:12:26.495 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:12:36.511 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:12:46.526 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:12:56.537 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:13:06.556 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:13:16.574 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:13:26.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:13:36.605 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:13:46.622 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T03:13:56.643 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:14:07.572 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:14:17.592 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:14:27.612 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:14:37.641 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:14:48.295 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:14:58.314 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:15:08.339 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:15:18.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T03:15:28.630 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuvalif@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633886'} 2020-11-19T03:15:28.651 DEBUG:teuthology.lock.ops:locked ubuntu@smithi066.front.sepia.ceph.com, ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:15:28.704 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efca7733950>, signals=[15]) 2020-11-19T03:15:28.920 INFO:teuthology.provision.fog.smithi066:Scheduling deploy of ubuntu 18.04 2020-11-19T03:15:28.924 INFO:teuthology.provision.fog.smithi154:Scheduling deploy of ubuntu 18.04 2020-11-19T03:15:29.330 INFO:teuthology.orchestra.console:Power off smithi154 2020-11-19T03:15:29.330 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-19T03:15:29.355 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:29.369 INFO:teuthology.orchestra.console:Power off smithi066 2020-11-19T03:15:29.370 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-19T03:15:29.390 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:33.379 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:33.414 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:37.492 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:37.524 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:37.703 INFO:teuthology.orchestra.console:Power off for smithi154 completed 2020-11-19T03:15:37.734 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2020-11-19T03:15:37.805 INFO:teuthology.orchestra.console:Power on smithi154 2020-11-19T03:15:37.806 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-19T03:15:37.828 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:37.838 INFO:teuthology.orchestra.console:Power on smithi066 2020-11-19T03:15:37.839 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-19T03:15:37.859 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:41.854 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:43.045 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-19T03:15:43.247 INFO:teuthology.orchestra.console:Power on for smithi154 completed 2020-11-19T03:15:43.255 INFO:teuthology.orchestra.console:Power on for smithi066 completed 2020-11-19T03:15:43.349 INFO:teuthology.provision.fog.smithi154:Waiting for deploy to finish 2020-11-19T03:15:43.356 INFO:teuthology.provision.fog.smithi066:Waiting for deploy to finish 2020-11-19T03:18:45.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:18:45.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:09.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:09.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:18.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:18.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:27.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:28.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:37.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:37.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:19:46.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:02.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:11.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:20.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:29.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:35.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:36.684 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-19T03:20:36.686 INFO:teuthology.orchestra.run.smithi066:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-19T03:20:37.059 INFO:teuthology.provision.fog.smithi066:Node is ready 2020-11-19T03:20:37.059 INFO:teuthology.orchestra.run.smithi066:> hostname 2020-11-19T03:20:37.107 INFO:teuthology.orchestra.run.smithi066.stdout:smithi066.front.sepia.ceph.com 2020-11-19T03:20:37.108 INFO:teuthology.orchestra.run.smithi066:> grep smithi066.front.sepia.ceph.com /etc/hosts 2020-11-19T03:20:37.157 INFO:teuthology.orchestra.run.smithi066.stdout:172.21.15.66 smithi066.front.sepia.ceph.com smithi066 2020-11-19T03:20:37.157 INFO:teuthology.orchestra.run.smithi066:> sudo hostname smithi066 2020-11-19T03:20:37.221 INFO:teuthology.orchestra.run.smithi066:> sudo sed -i -e 's/smithi066.front.sepia.ceph.com/smithi066/g' /etc/hosts 2020-11-19T03:20:37.282 INFO:teuthology.orchestra.run.smithi066:> sudo sed -i -e 's/172.21.15.66/172.21.15.66/g' /etc/hosts 2020-11-19T03:20:37.339 INFO:teuthology.orchestra.run.smithi066:> sudo sed -i -e 's/smithi066.front.sepia.ceph.com/smithi066/g' /etc/hostname 2020-11-19T03:20:37.395 INFO:teuthology.orchestra.run.smithi066:> sudo hostname smithi066 2020-11-19T03:20:37.451 INFO:teuthology.provision.fog.smithi066:Deploy complete! 2020-11-19T03:20:37.452 DEBUG:teuthology.parallel:result is None 2020-11-19T03:20:52.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:53.032 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-19T03:20:53.032 INFO:teuthology.orchestra.run.smithi154:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-19T03:20:53.394 INFO:teuthology.provision.fog.smithi154:Node is ready 2020-11-19T03:20:53.395 INFO:teuthology.orchestra.run.smithi154:> hostname 2020-11-19T03:20:53.442 INFO:teuthology.orchestra.run.smithi154.stdout:smithi154.front.sepia.ceph.com 2020-11-19T03:20:53.443 INFO:teuthology.orchestra.run.smithi154:> grep smithi154.front.sepia.ceph.com /etc/hosts 2020-11-19T03:20:53.491 INFO:teuthology.orchestra.run.smithi154.stdout:172.21.15.154 smithi154.front.sepia.ceph.com smithi154 2020-11-19T03:20:53.491 INFO:teuthology.orchestra.run.smithi154:> sudo hostname smithi154 2020-11-19T03:20:53.597 INFO:teuthology.orchestra.run.smithi154:> sudo sed -i -e 's/smithi154.front.sepia.ceph.com/smithi154/g' /etc/hosts 2020-11-19T03:20:53.611 INFO:teuthology.orchestra.run.smithi154:> sudo sed -i -e 's/172.21.15.154/172.21.15.154/g' /etc/hosts 2020-11-19T03:20:53.668 INFO:teuthology.orchestra.run.smithi154:> sudo sed -i -e 's/smithi154.front.sepia.ceph.com/smithi154/g' /etc/hostname 2020-11-19T03:20:53.724 INFO:teuthology.orchestra.run.smithi154:> sudo hostname smithi154 2020-11-19T03:20:53.779 INFO:teuthology.provision.fog.smithi154:Deploy complete! 2020-11-19T03:20:53.780 DEBUG:teuthology.parallel:result is None 2020-11-19T03:20:53.907 INFO:teuthology.lock.ops:Checking smithi066.front.sepia.ceph.com 2020-11-19T03:20:53.908 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-19T03:20:53.925 INFO:teuthology.lock.ops:Checking smithi154.front.sepia.ceph.com 2020-11-19T03:20:53.925 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-19T03:20:53.940 INFO:teuthology.lock.ops:Updating [smithi066.front.sepia.ceph.com]: set os type and version on server 2020-11-19T03:20:53.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:20:53.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:54.018 INFO:teuthology.orchestra.run.smithi066:> true 2020-11-19T03:20:54.267 INFO:teuthology.orchestra.run.smithi066:> uname -m 2020-11-19T03:20:54.315 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2020-11-19T03:20:54.315 INFO:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2020-11-19T03:20:54.362 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="Ubuntu" 2020-11-19T03:20:54.363 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T03:20:54.363 INFO:teuthology.orchestra.run.smithi066.stdout:ID=ubuntu 2020-11-19T03:20:54.363 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE=debian 2020-11-19T03:20:54.364 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T03:20:54.364 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="18.04" 2020-11-19T03:20:54.364 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T03:20:54.364 INFO:teuthology.orchestra.run.smithi066.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T03:20:54.364 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T03:20:54.365 INFO:teuthology.orchestra.run.smithi066.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T03:20:54.365 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_CODENAME=bionic 2020-11-19T03:20:54.365 INFO:teuthology.orchestra.run.smithi066.stdout:UBUNTU_CODENAME=bionic 2020-11-19T03:20:54.366 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2020-11-19T03:20:54.382 INFO:teuthology.lock.ops:Updating [smithi154.front.sepia.ceph.com]: set os type and version on server 2020-11-19T03:20:54.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:20:54.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:54.459 INFO:teuthology.orchestra.run.smithi154:> true 2020-11-19T03:20:54.703 INFO:teuthology.orchestra.run.smithi154:> uname -m 2020-11-19T03:20:54.751 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2020-11-19T03:20:54.752 INFO:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2020-11-19T03:20:54.799 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2020-11-19T03:20:54.799 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T03:20:54.800 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2020-11-19T03:20:54.800 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2020-11-19T03:20:54.800 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T03:20:54.801 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="18.04" 2020-11-19T03:20:54.801 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T03:20:54.801 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T03:20:54.802 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T03:20:54.802 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T03:20:54.803 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=bionic 2020-11-19T03:20:54.803 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=bionic 2020-11-19T03:20:54.804 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2020-11-19T03:20:54.822 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-19T03:20:54.869 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcRZ1FksMeIk6CFCb3W6w2tfikSMp9EDkwlSGqUYBi2RLjZJn2SE1rITizpE4X5hGFzRU8SdaWIy3iuhD4VIPrEIa7/SFpk9OfJzkorD6I72swHFJKBahLLH6aJX5Yx8/I7BLiYXzJ0+xyEfobvaZZVf/7gyHldHTDYOU9Lls45PF0S/RN/ZXE+S16ME6+Ef0pIfkwBENcFDcKBdFcL5pk0z6jedB7GMwwrXsGd6EBcOUYxhR6y+wxkMgHMSBVyflVwTK85snOI6shbg9PrvqdnZRkv24i1Kc2lE33YwhXgnp1lLhSceER2/FXlgmz2th2cIsnFOCDY7u12H8T+knj smithi154.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5EGCCacrU4DTfSf3En71+Jcujq6iYhrNBfANMGFlLcrKHyhaRuoNs89UKHMHPSgyJL+hi/5JStDf/DpnvV3aXcLCOL414b4ohNF0Gfrc6PGDmt3WNAjLNr3pweEUgkYpyRFH4f5snFluElHQJ+KdwuU6GAS/c/a87rPkLUAyCa0LaxR5x3dMC+MXH7nRmaOITXjAQNO8e5u1AZPubTFb73nWWHmeBD3tSav1CVI8x1zYgVIfIDrHTeLQwy4dviozplDeUZP/GL0yvq2RxTEbUdWhytvVgnEyCCcuyXmAQaaCc+K8k0Rsw3UqUnhNr1XV/dqaxHFKeW61wcgo1B8GR 2020-11-19T03:20:54.869 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T03:20:54.921 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-19T03:20:54.930 INFO:teuthology.task.internal:Saving configuration 2020-11-19T03:20:54.946 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-19T03:20:54.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-19T03:20:54.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-19 03:15:28.635447', 'description': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633886', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIyugmdPM9+VGzTtFW/Wq+4yW6mC8ftSVOMZSGeS1jCs', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi066.front.sepia.ceph.com'} 2020-11-19T03:20:54.992 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-19 03:15:28.635470', 'description': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633886', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKUbLeFOw+3isui2C0f21znyfNeaQt4nwKHnlNDbJ6yz', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi154.front.sepia.ceph.com'} 2020-11-19T03:20:54.992 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-19T03:20:55.003 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-11-19T03:20:55.003 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-11-19T03:20:55.003 INFO:teuthology.run_tasks:Running task console_log... 2020-11-19T03:20:55.069 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efcab25fe18>, signals=[15]) 2020-11-19T03:20:55.070 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-19T03:20:55.081 INFO:teuthology.task.internal:Opening connections... 2020-11-19T03:20:55.082 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:20:55.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:55.161 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:20:55.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:20:55.328 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-19T03:20:55.339 INFO:teuthology.orchestra.run.smithi066:> uname -m 2020-11-19T03:20:55.365 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2020-11-19T03:20:55.366 INFO:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2020-11-19T03:20:55.423 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="Ubuntu" 2020-11-19T03:20:55.423 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T03:20:55.423 INFO:teuthology.orchestra.run.smithi066.stdout:ID=ubuntu 2020-11-19T03:20:55.424 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE=debian 2020-11-19T03:20:55.424 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T03:20:55.424 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="18.04" 2020-11-19T03:20:55.424 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T03:20:55.425 INFO:teuthology.orchestra.run.smithi066.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T03:20:55.425 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T03:20:55.425 INFO:teuthology.orchestra.run.smithi066.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T03:20:55.425 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_CODENAME=bionic 2020-11-19T03:20:55.426 INFO:teuthology.orchestra.run.smithi066.stdout:UBUNTU_CODENAME=bionic 2020-11-19T03:20:55.426 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2020-11-19T03:20:55.443 INFO:teuthology.orchestra.run.smithi154:> uname -m 2020-11-19T03:20:55.527 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2020-11-19T03:20:55.528 INFO:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2020-11-19T03:20:55.577 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2020-11-19T03:20:55.577 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-19T03:20:55.578 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2020-11-19T03:20:55.578 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2020-11-19T03:20:55.578 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-19T03:20:55.578 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="18.04" 2020-11-19T03:20:55.579 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-19T03:20:55.579 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-19T03:20:55.579 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-19T03:20:55.579 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-19T03:20:55.579 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=bionic 2020-11-19T03:20:55.580 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=bionic 2020-11-19T03:20:55.580 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2020-11-19T03:20:55.603 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-19T03:20:55.759 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-19T03:20:55.768 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-19T03:20:55.768 INFO:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2020-11-19T03:20:55.772 INFO:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2020-11-19T03:20:55.777 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-19T03:20:55.826 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-19T03:20:55.827 INFO:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2020-11-19T03:20:55.829 INFO:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2020-11-19T03:20:55.852 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-19T03:20:55.901 INFO:teuthology.run_tasks:Running task kernel... 2020-11-19T03:20:55.917 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-11-19T03:20:55.918 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-19T03:20:55.918 INFO:teuthology.orchestra.run.smithi066:> uname -r 2020-11-19T03:20:55.925 INFO:teuthology.orchestra.run.smithi066.stdout:4.15.0-123-generic 2020-11-19T03:20:55.925 INFO:teuthology.task.kernel:Running kernel on smithi066: 4.15.0-123-generic 2020-11-19T03:20:55.925 INFO:teuthology.orchestra.run.smithi066:> sudo apt-get clean 2020-11-19T03:20:56.037 INFO:teuthology.orchestra.run.smithi066:> sudo apt-get update 2020-11-19T03:20:56.306 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T03:20:56.307 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-19T03:20:56.368 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-19T03:20:56.622 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-19T03:20:58.890 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [861 kB] 2020-11-19T03:20:58.891 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,761 kB] 2020-11-19T03:20:58.891 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,426 kB] 2020-11-19T03:20:58.891 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,162 kB] 2020-11-19T03:20:58.891 INFO:teuthology.orchestra.run.smithi066.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [966 kB] 2020-11-19T03:20:58.891 INFO:teuthology.orchestra.run.smithi066.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [373 kB] 2020-11-19T03:20:58.892 INFO:teuthology.orchestra.run.smithi066.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,088 kB] 2020-11-19T03:20:58.892 INFO:teuthology.orchestra.run.smithi066.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.3 kB] 2020-11-19T03:20:58.892 INFO:teuthology.orchestra.run.smithi066.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [196 kB] 2020-11-19T03:20:58.892 INFO:teuthology.orchestra.run.smithi066.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [26.3 kB] 2020-11-19T03:20:58.892 INFO:teuthology.orchestra.run.smithi066.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-19T03:20:58.893 INFO:teuthology.orchestra.run.smithi066.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,547 kB] 2020-11-19T03:20:58.893 INFO:teuthology.orchestra.run.smithi066.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [355 kB] 2020-11-19T03:20:58.894 INFO:teuthology.orchestra.run.smithi066.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [36.1 kB] 2020-11-19T03:20:58.894 INFO:teuthology.orchestra.run.smithi066.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.4 kB] 2020-11-19T03:20:58.894 INFO:teuthology.orchestra.run.smithi066.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,308 B] 2020-11-19T03:20:58.895 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 11.8 MB in 2s (4,884 kB/s) 2020-11-19T03:21:00.146 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:21:00.194 INFO:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T03:21:00.279 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:21:00.518 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2020-11-19T03:21:00.518 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2020-11-19T03:21:00.602 INFO:teuthology.orchestra.run.smithi066.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T03:21:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T03:21:00.604 INFO:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T03:21:00.694 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:21:00.930 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2020-11-19T03:21:00.930 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2020-11-19T03:21:01.112 INFO:teuthology.orchestra.run.smithi066.stdout:The following additional packages will be installed: 2020-11-19T03:21:01.113 INFO:teuthology.orchestra.run.smithi066.stdout: linux-generic linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T03:21:01.113 INFO:teuthology.orchestra.run.smithi066.stdout: linux-headers-generic linux-image-4.15.0-124-generic 2020-11-19T03:21:01.114 INFO:teuthology.orchestra.run.smithi066.stdout: linux-modules-4.15.0-124-generic linux-modules-extra-4.15.0-124-generic 2020-11-19T03:21:01.115 INFO:teuthology.orchestra.run.smithi066.stdout:Suggested packages: 2020-11-19T03:21:01.115 INFO:teuthology.orchestra.run.smithi066.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-19T03:21:01.115 INFO:teuthology.orchestra.run.smithi066.stdout:Recommended packages: 2020-11-19T03:21:01.116 INFO:teuthology.orchestra.run.smithi066.stdout: thermald 2020-11-19T03:21:01.142 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2020-11-19T03:21:01.142 INFO:teuthology.orchestra.run.smithi066.stdout: linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T03:21:01.143 INFO:teuthology.orchestra.run.smithi066.stdout: linux-image-4.15.0-124-generic linux-modules-4.15.0-124-generic 2020-11-19T03:21:01.143 INFO:teuthology.orchestra.run.smithi066.stdout: linux-modules-extra-4.15.0-124-generic 2020-11-19T03:21:01.144 INFO:teuthology.orchestra.run.smithi066.stdout:The following packages will be upgraded: 2020-11-19T03:21:01.145 INFO:teuthology.orchestra.run.smithi066.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-19T03:21:01.322 INFO:teuthology.orchestra.run.smithi066.stdout:3 upgraded, 5 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T03:21:01.322 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 65.9 MB of archives. 2020-11-19T03:21:01.323 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-19T03:21:01.323 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-124-generic amd64 4.15.0-124.127 [13.1 MB] 2020-11-19T03:21:02.310 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-124-generic amd64 4.15.0-124.127 [8,012 kB] 2020-11-19T03:21:02.650 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-124-generic amd64 4.15.0-124.127 [32.8 MB] 2020-11-19T03:21:03.501 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.124.111 [1,868 B] 2020-11-19T03:21:03.502 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.124.111 [2,624 B] 2020-11-19T03:21:03.502 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124 all 4.15.0-124.127 [10.9 MB] 2020-11-19T03:21:03.810 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124-generic amd64 4.15.0-124.127 [1,084 kB] 2020-11-19T03:21:03.830 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.124.111 [2,520 B] 2020-11-19T03:21:04.321 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 65.9 MB in 3s (24.7 MB/s) 2020-11-19T03:21:04.486 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-modules-4.15.0-124-generic. 2020-11-19T03:21:04.525 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-19T03:21:04.528 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../0-linux-modules-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:21:04.562 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:06.238 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-image-4.15.0-124-generic. 2020-11-19T03:21:06.250 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../1-linux-image-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:21:06.338 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:06.816 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-124-generic. 2020-11-19T03:21:06.829 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:21:06.857 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:11.015 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../3-linux-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T03:21:11.137 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T03:21:11.433 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T03:21:11.538 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-image-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T03:21:11.866 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-headers-4.15.0-124. 2020-11-19T03:21:11.878 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../5-linux-headers-4.15.0-124_4.15.0-124.127_all.deb ... 2020-11-19T03:21:11.916 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T03:21:16.723 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-headers-4.15.0-124-generic. 2020-11-19T03:21:16.746 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../6-linux-headers-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:21:16.790 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:19.205 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T03:21:19.327 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-headers-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T03:21:19.641 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:20.205 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:20.626 INFO:teuthology.orchestra.run.smithi066.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-124-generic 2020-11-19T03:21:20.627 INFO:teuthology.orchestra.run.smithi066.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-124-generic 2020-11-19T03:21:20.749 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:21.250 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T03:21:21.376 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-image-generic (4.15.0.124.111) ... 2020-11-19T03:21:21.502 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:21.628 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-headers-generic (4.15.0.124.111) ... 2020-11-19T03:21:21.771 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-generic (4.15.0.124.111) ... 2020-11-19T03:21:21.880 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:22.085 INFO:teuthology.orchestra.run.smithi066.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-11-19T03:21:22.086 INFO:teuthology.orchestra.run.smithi066.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-124-generic 2020-11-19T03:21:37.762 INFO:teuthology.orchestra.run.smithi066.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-11-19T03:21:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:Sourcing file `/etc/default/grub' 2020-11-19T03:21:37.791 INFO:teuthology.orchestra.run.smithi066.stdout:Generating grub configuration file ... 2020-11-19T03:21:38.402 INFO:teuthology.orchestra.run.smithi066.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T03:21:38.435 INFO:teuthology.orchestra.run.smithi066.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T03:21:38.772 INFO:teuthology.orchestra.run.smithi066.stdout:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T03:21:38.801 INFO:teuthology.orchestra.run.smithi066.stdout:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T03:21:40.204 INFO:teuthology.orchestra.run.smithi066.stdout:done 2020-11-19T03:21:40.621 INFO:teuthology.orchestra.run.smithi066:> dpkg -s linux-image-generic 2020-11-19T03:21:40.648 INFO:teuthology.orchestra.run.smithi066.stdout:Package: linux-image-generic 2020-11-19T03:21:40.649 INFO:teuthology.orchestra.run.smithi066.stdout:Status: install ok installed 2020-11-19T03:21:40.649 INFO:teuthology.orchestra.run.smithi066.stdout:Priority: optional 2020-11-19T03:21:40.649 INFO:teuthology.orchestra.run.smithi066.stdout:Section: kernel 2020-11-19T03:21:40.650 INFO:teuthology.orchestra.run.smithi066.stdout:Installed-Size: 16 2020-11-19T03:21:40.650 INFO:teuthology.orchestra.run.smithi066.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T03:21:40.650 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: amd64 2020-11-19T03:21:40.650 INFO:teuthology.orchestra.run.smithi066.stdout:Source: linux-meta 2020-11-19T03:21:40.650 INFO:teuthology.orchestra.run.smithi066.stdout:Version: 4.15.0.124.111 2020-11-19T03:21:40.651 INFO:teuthology.orchestra.run.smithi066.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T03:21:40.651 INFO:teuthology.orchestra.run.smithi066.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T03:21:40.651 INFO:teuthology.orchestra.run.smithi066.stdout:Recommends: thermald 2020-11-19T03:21:40.651 INFO:teuthology.orchestra.run.smithi066.stdout:Description: Generic Linux kernel image 2020-11-19T03:21:40.651 INFO:teuthology.orchestra.run.smithi066.stdout: This package will always depend on the latest generic kernel image 2020-11-19T03:21:40.652 INFO:teuthology.orchestra.run.smithi066.stdout: available. 2020-11-19T03:21:40.652 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-123-generic Expected: 4.15.0-124-generic 2020-11-19T03:21:40.652 INFO:teuthology.orchestra.run.smithi154:> uname -r 2020-11-19T03:21:40.662 INFO:teuthology.orchestra.run.smithi154.stdout:4.15.0-123-generic 2020-11-19T03:21:40.663 INFO:teuthology.task.kernel:Running kernel on smithi154: 4.15.0-123-generic 2020-11-19T03:21:40.663 INFO:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2020-11-19T03:21:40.804 INFO:teuthology.orchestra.run.smithi154:> sudo apt-get update 2020-11-19T03:21:40.922 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-19T03:21:41.046 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T03:21:41.126 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-19T03:21:41.185 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [861 kB] 2020-11-19T03:21:41.272 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,426 kB] 2020-11-19T03:21:41.297 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [966 kB] 2020-11-19T03:21:41.308 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,088 kB] 2020-11-19T03:21:41.389 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-19T03:21:41.513 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,761 kB] 2020-11-19T03:21:41.838 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,162 kB] 2020-11-19T03:21:41.892 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [373 kB] 2020-11-19T03:21:41.907 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [196 kB] 2020-11-19T03:21:41.914 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.3 kB] 2020-11-19T03:21:41.914 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [26.3 kB] 2020-11-19T03:21:41.915 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-19T03:21:41.956 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,547 kB] 2020-11-19T03:21:42.005 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [355 kB] 2020-11-19T03:21:42.010 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [36.1 kB] 2020-11-19T03:21:42.010 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.4 kB] 2020-11-19T03:21:42.011 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,308 B] 2020-11-19T03:21:42.908 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 11.8 MB in 2s (6,305 kB/s) 2020-11-19T03:21:44.339 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:21:44.386 INFO:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T03:21:44.465 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:21:44.705 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2020-11-19T03:21:44.706 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2020-11-19T03:21:44.784 INFO:teuthology.orchestra.run.smithi154.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T03:21:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T03:21:44.786 INFO:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T03:21:44.867 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:21:45.107 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2020-11-19T03:21:45.107 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2020-11-19T03:21:45.282 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2020-11-19T03:21:45.283 INFO:teuthology.orchestra.run.smithi154.stdout: linux-generic linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T03:21:45.284 INFO:teuthology.orchestra.run.smithi154.stdout: linux-headers-generic linux-image-4.15.0-124-generic 2020-11-19T03:21:45.284 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-4.15.0-124-generic linux-modules-extra-4.15.0-124-generic 2020-11-19T03:21:45.287 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2020-11-19T03:21:45.287 INFO:teuthology.orchestra.run.smithi154.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-19T03:21:45.288 INFO:teuthology.orchestra.run.smithi154.stdout:Recommended packages: 2020-11-19T03:21:45.288 INFO:teuthology.orchestra.run.smithi154.stdout: thermald 2020-11-19T03:21:45.309 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2020-11-19T03:21:45.310 INFO:teuthology.orchestra.run.smithi154.stdout: linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-19T03:21:45.310 INFO:teuthology.orchestra.run.smithi154.stdout: linux-image-4.15.0-124-generic linux-modules-4.15.0-124-generic 2020-11-19T03:21:45.310 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-extra-4.15.0-124-generic 2020-11-19T03:21:45.311 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages will be upgraded: 2020-11-19T03:21:45.312 INFO:teuthology.orchestra.run.smithi154.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-19T03:21:45.490 INFO:teuthology.orchestra.run.smithi154.stdout:3 upgraded, 5 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T03:21:45.490 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 65.9 MB of archives. 2020-11-19T03:21:45.491 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-19T03:21:45.491 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-124-generic amd64 4.15.0-124.127 [13.1 MB] 2020-11-19T03:21:46.510 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-124-generic amd64 4.15.0-124.127 [8,012 kB] 2020-11-19T03:21:46.715 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-124-generic amd64 4.15.0-124.127 [32.8 MB] 2020-11-19T03:21:47.706 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.124.111 [1,868 B] 2020-11-19T03:21:47.706 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.124.111 [2,624 B] 2020-11-19T03:21:47.706 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124 all 4.15.0-124.127 [10.9 MB] 2020-11-19T03:21:48.024 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124-generic amd64 4.15.0-124.127 [1,084 kB] 2020-11-19T03:21:48.065 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.124.111 [2,520 B] 2020-11-19T03:21:48.514 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 65.9 MB in 3s (24.1 MB/s) 2020-11-19T03:21:48.679 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-4.15.0-124-generic. 2020-11-19T03:21:48.717 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-19T03:21:48.720 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../0-linux-modules-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:21:48.764 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:50.381 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-image-4.15.0-124-generic. 2020-11-19T03:21:50.394 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../1-linux-image-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:21:50.488 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:51.001 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-124-generic. 2020-11-19T03:21:51.014 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:21:51.054 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:21:54.752 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../3-linux-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T03:21:54.880 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T03:21:55.197 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T03:21:55.342 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T03:21:55.669 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-headers-4.15.0-124. 2020-11-19T03:21:55.718 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../5-linux-headers-4.15.0-124_4.15.0-124.127_all.deb ... 2020-11-19T03:21:55.719 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T03:22:00.379 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-headers-4.15.0-124-generic. 2020-11-19T03:22:00.401 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../6-linux-headers-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-19T03:22:00.445 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:22:02.738 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.124.111_amd64.deb ... 2020-11-19T03:22:02.862 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-19T03:22:03.203 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:22:03.791 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:22:04.241 INFO:teuthology.orchestra.run.smithi154.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-124-generic 2020-11-19T03:22:04.242 INFO:teuthology.orchestra.run.smithi154.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-124-generic 2020-11-19T03:22:04.743 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:22:05.227 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-19T03:22:05.353 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-generic (4.15.0.124.111) ... 2020-11-19T03:22:05.479 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:22:05.605 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-generic (4.15.0.124.111) ... 2020-11-19T03:22:05.731 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-generic (4.15.0.124.111) ... 2020-11-19T03:22:05.860 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-19T03:22:06.066 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-11-19T03:22:06.067 INFO:teuthology.orchestra.run.smithi154.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-124-generic 2020-11-19T03:22:20.858 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-11-19T03:22:20.859 INFO:teuthology.orchestra.run.smithi154.stdout:Sourcing file `/etc/default/grub' 2020-11-19T03:22:20.898 INFO:teuthology.orchestra.run.smithi154.stdout:Generating grub configuration file ... 2020-11-19T03:22:21.272 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T03:22:21.303 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T03:22:21.629 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T03:22:21.658 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T03:22:23.356 INFO:teuthology.orchestra.run.smithi154.stdout:done 2020-11-19T03:22:24.762 INFO:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic 2020-11-19T03:22:24.788 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic 2020-11-19T03:22:24.788 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2020-11-19T03:22:24.789 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2020-11-19T03:22:24.789 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2020-11-19T03:22:24.789 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 16 2020-11-19T03:22:24.789 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T03:22:24.789 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2020-11-19T03:22:24.790 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta 2020-11-19T03:22:24.790 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 4.15.0.124.111 2020-11-19T03:22:24.790 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T03:22:24.790 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T03:22:24.791 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2020-11-19T03:22:24.791 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2020-11-19T03:22:24.791 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2020-11-19T03:22:24.791 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2020-11-19T03:22:24.791 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-123-generic Expected: 4.15.0-124-generic 2020-11-19T03:22:24.792 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-11-19T03:22:24.792 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-11-19T03:22:24.792 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-11-19T03:22:24.793 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-11-19T03:22:24.793 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2020-11-19T03:22:24.793 INFO:teuthology.orchestra.run.smithi066:> sudo apt-get clean 2020-11-19T03:22:24.854 INFO:teuthology.orchestra.run.smithi066:> sudo apt-get update 2020-11-19T03:22:25.103 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T03:22:25.104 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T03:22:25.170 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T03:22:25.250 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T03:22:27.018 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:22:27.068 INFO:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T03:22:27.147 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:22:27.397 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2020-11-19T03:22:27.398 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2020-11-19T03:22:27.483 INFO:teuthology.orchestra.run.smithi066.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T03:22:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T03:22:27.485 INFO:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T03:22:27.559 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:22:27.796 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2020-11-19T03:22:27.797 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2020-11-19T03:22:28.005 INFO:teuthology.orchestra.run.smithi066.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-19T03:22:28.005 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T03:22:28.006 INFO:teuthology.orchestra.run.smithi066:> dpkg -s linux-image-generic 2020-11-19T03:22:28.042 INFO:teuthology.orchestra.run.smithi066.stdout:Package: linux-image-generic 2020-11-19T03:22:30.059 INFO:teuthology.orchestra.run.smithi066.stdout:Status: install ok installed 2020-11-19T03:22:30.059 INFO:teuthology.orchestra.run.smithi066.stdout:Priority: optional 2020-11-19T03:22:30.059 INFO:teuthology.orchestra.run.smithi066.stdout:Section: kernel 2020-11-19T03:22:30.060 INFO:teuthology.orchestra.run.smithi066.stdout:Installed-Size: 16 2020-11-19T03:22:30.060 INFO:teuthology.orchestra.run.smithi066.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T03:22:30.060 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: amd64 2020-11-19T03:22:30.060 INFO:teuthology.orchestra.run.smithi066.stdout:Source: linux-meta 2020-11-19T03:22:30.060 INFO:teuthology.orchestra.run.smithi066.stdout:Version: 4.15.0.124.111 2020-11-19T03:22:30.061 INFO:teuthology.orchestra.run.smithi066.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T03:22:30.061 INFO:teuthology.orchestra.run.smithi066.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T03:22:30.061 INFO:teuthology.orchestra.run.smithi066.stdout:Recommends: thermald 2020-11-19T03:22:30.061 INFO:teuthology.orchestra.run.smithi066.stdout:Description: Generic Linux kernel image 2020-11-19T03:22:30.061 INFO:teuthology.orchestra.run.smithi066.stdout: This package will always depend on the latest generic kernel image 2020-11-19T03:22:30.062 INFO:teuthology.orchestra.run.smithi066.stdout: available. 2020-11-19T03:22:30.062 INFO:teuthology.orchestra.run.smithi066:> mktemp 2020-11-19T03:22:30.071 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.ECnx3t8H7O 2020-11-19T03:22:30.072 INFO:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ECnx3t8H7O 2020-11-19T03:22:30.128 INFO:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.ECnx3t8H7O 2020-11-19T03:22:30.241 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.ECnx3t8H7O is 8KB 2020-11-19T03:22:30.302 INFO:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.ECnx3t8H7O 2020-11-19T03:22:30.309 INFO:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-11-19T03:22:30.367 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:22:30.367 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T03:22:30.368 INFO:teuthology.orchestra.run.smithi066:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T03:22:30.413 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T03:22:30.428 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:22:30.429 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:22:30.429 INFO:teuthology.orchestra.run.smithi066.stderr:93 bytes copied, 3.4657e-05 s, 2.7 MB/s 2020-11-19T03:22:30.429 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T03:22:30.440 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-124-generic 2020-11-19T03:22:30.441 INFO:teuthology.orchestra.run.smithi066:> sudo update-grub 2020-11-19T03:22:31.674 INFO:teuthology.orchestra.run.smithi066.stderr:Sourcing file `/etc/default/grub' 2020-11-19T03:22:31.692 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2020-11-19T03:22:32.023 INFO:teuthology.orchestra.run.smithi066.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T03:22:32.035 INFO:teuthology.orchestra.run.smithi066.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T03:22:32.336 INFO:teuthology.orchestra.run.smithi066.stderr:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T03:22:32.350 INFO:teuthology.orchestra.run.smithi066.stderr:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T03:22:33.473 INFO:teuthology.orchestra.run.smithi066.stderr:done 2020-11-19T03:22:33.731 INFO:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2020-11-19T03:22:33.748 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-11-19T03:22:33.749 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2020-11-19T03:22:33.749 INFO:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2020-11-19T03:22:33.813 INFO:teuthology.orchestra.run.smithi154:> sudo apt-get update 2020-11-19T03:22:33.943 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T03:22:34.055 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T03:22:34.133 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T03:22:34.224 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T03:22:35.812 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:22:35.854 INFO:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-19T03:22:35.928 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:22:36.172 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2020-11-19T03:22:36.173 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2020-11-19T03:22:36.251 INFO:teuthology.orchestra.run.smithi154.stderr:E: Unable to locate package linux-image-current-generic 2020-11-19T03:22:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-19T03:22:36.253 INFO:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-19T03:22:36.327 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:22:36.556 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2020-11-19T03:22:36.557 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2020-11-19T03:22:36.751 INFO:teuthology.orchestra.run.smithi154.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-19T03:22:36.752 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T03:22:36.752 INFO:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic 2020-11-19T03:22:36.770 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic 2020-11-19T03:22:36.770 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2020-11-19T03:22:36.770 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2020-11-19T03:22:36.770 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2020-11-19T03:22:36.771 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 16 2020-11-19T03:22:36.771 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2020-11-19T03:22:36.771 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2020-11-19T03:22:36.771 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta 2020-11-19T03:22:36.771 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 4.15.0.124.111 2020-11-19T03:22:36.772 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-19T03:22:36.772 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-19T03:22:36.772 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2020-11-19T03:22:36.772 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2020-11-19T03:22:36.773 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2020-11-19T03:22:36.773 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2020-11-19T03:22:36.773 INFO:teuthology.orchestra.run.smithi154:> mktemp 2020-11-19T03:22:36.815 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.33nYiafwHs 2020-11-19T03:22:36.816 INFO:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub/grub.cfg /tmp/tmp.33nYiafwHs 2020-11-19T03:22:36.891 INFO:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.33nYiafwHs 2020-11-19T03:22:37.010 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.33nYiafwHs is 8KB 2020-11-19T03:22:37.067 INFO:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.33nYiafwHs 2020-11-19T03:22:37.082 INFO:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-11-19T03:22:37.147 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:22:37.149 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T03:22:37.150 INFO:teuthology.orchestra.run.smithi154:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T03:22:37.197 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-19T03:22:37.245 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:22:37.245 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:22:37.246 INFO:teuthology.orchestra.run.smithi154.stderr:93 bytes copied, 0.0351838 s, 2.6 kB/s 2020-11-19T03:22:37.249 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-19T03:22:37.258 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-124-generic 2020-11-19T03:22:37.259 INFO:teuthology.orchestra.run.smithi154:> sudo update-grub 2020-11-19T03:22:38.628 INFO:teuthology.orchestra.run.smithi154.stderr:Sourcing file `/etc/default/grub' 2020-11-19T03:22:40.607 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2020-11-19T03:22:40.608 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-19T03:22:40.608 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-19T03:22:40.608 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-19T03:22:40.608 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-19T03:22:40.608 INFO:teuthology.orchestra.run.smithi154.stderr:done 2020-11-19T03:22:40.609 INFO:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2020-11-19T03:23:10.638 INFO:teuthology.misc:Re-opening connections... 2020-11-19T03:23:10.639 INFO:teuthology.misc:trying to connect to ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:23:10.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:23:10.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:24:10.643 DEBUG:teuthology.orchestra.remote:timed out 2020-11-19T03:24:10.649 INFO:teuthology.misc:trying to connect to ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:24:10.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:24:10.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:24:27.421 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2020-11-19T03:24:27.421 DEBUG:teuthology.misc:waited 76.78244543075562 2020-11-19T03:24:28.424 INFO:teuthology.misc:trying to connect to ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:24:28.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:24:28.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:24:28.522 INFO:teuthology.orchestra.run.smithi066:> true 2020-11-19T03:24:29.011 DEBUG:teuthology.misc:waited 78.37175130844116 2020-11-19T03:24:30.012 INFO:teuthology.misc:trying to connect to ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:24:30.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:24:30.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:24:30.153 INFO:teuthology.orchestra.run.smithi154:> true 2020-11-19T03:24:30.523 DEBUG:teuthology.misc:waited 79.8836784362793 2020-11-19T03:24:31.524 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-11-19T03:24:31.524 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-124-generic"... 2020-11-19T03:24:31.525 INFO:teuthology.orchestra.run.smithi066:> uname -r 2020-11-19T03:24:31.537 INFO:teuthology.orchestra.run.smithi066.stdout:4.15.0-124-generic 2020-11-19T03:24:31.540 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-124-generic vs 4.15.0-124-generic 2020-11-19T03:24:31.541 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-19T03:24:31.542 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-11-19T03:24:31.543 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-11-19T03:24:31.544 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-124-generic"... 2020-11-19T03:24:31.545 INFO:teuthology.orchestra.run.smithi154:> uname -r 2020-11-19T03:24:31.557 INFO:teuthology.orchestra.run.smithi154.stdout:4.15.0-124-generic 2020-11-19T03:24:31.558 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-124-generic vs 4.15.0-124-generic 2020-11-19T03:24:31.558 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-19T03:24:31.558 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-11-19T03:24:32.560 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-11-19T03:24:32.560 INFO:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-19T03:24:32.602 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2020-11-19T03:24:32.633 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-11-19T03:24:32.634 INFO:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-19T03:24:32.675 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2020-11-19T03:24:32.695 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-19T03:24:32.700 INFO:teuthology.task.internal:Creating test directory... 2020-11-19T03:24:32.701 INFO:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-19T03:24:32.703 INFO:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-19T03:24:32.753 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-19T03:24:32.771 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-19T03:24:32.867 INFO:teuthology.task.internal:Creating archive directory... 2020-11-19T03:24:33.526 INFO:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-19T03:24:33.530 INFO:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-19T03:24:33.550 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-19T03:24:33.560 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-19T03:24:33.561 INFO:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-19T03:24:33.585 INFO:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-19T03:24:33.609 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T03:24:33.616 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T03:24:33.620 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T03:24:33.624 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-19T03:24:33.625 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-19T03:24:33.652 INFO:teuthology.task.internal:Configuring sudo... 2020-11-19T03:24:33.652 INFO:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-19T03:24:33.669 INFO:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-19T03:24:33.687 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-19T03:24:34.067 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-19T03:24:34.068 INFO:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-19T03:24:34.070 INFO:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-19T03:24:34.077 INFO:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-19T03:24:34.125 INFO:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-19T03:24:34.174 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:24:34.174 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T03:24:34.220 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T03:24:34.238 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:24:34.238 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:24:34.238 INFO:teuthology.orchestra.run.smithi066.stderr:152 bytes copied, 3.2863e-05 s, 4.6 MB/s 2020-11-19T03:24:34.239 INFO:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-19T03:24:34.246 INFO:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-19T03:24:34.295 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:24:34.295 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T03:24:34.341 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-19T03:24:34.355 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:24:34.356 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:24:34.356 INFO:teuthology.orchestra.run.smithi154.stderr:152 bytes copied, 3.2748e-05 s, 4.6 MB/s 2020-11-19T03:24:34.357 INFO:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2020-11-19T03:24:34.359 INFO:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2020-11-19T03:24:34.443 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-19T03:24:34.550 INFO:teuthology.task.internal:Starting timer... 2020-11-19T03:24:34.551 INFO:teuthology.run_tasks:Running task pcp... 2020-11-19T03:24:34.582 INFO:teuthology.run_tasks:Running task selinux... 2020-11-19T03:24:34.603 DEBUG:teuthology.task.selinux:Excluding smithi066: OS 'ubuntu' does not support SELinux 2020-11-19T03:24:34.603 DEBUG:teuthology.task.selinux:Excluding smithi154: OS 'ubuntu' does not support SELinux 2020-11-19T03:24:34.604 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-19T03:24:34.604 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-19T03:24:34.604 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-19T03:24:34.604 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-19T03:24:34.616 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-19T03:24:34.626 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-19T03:24:34.715 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-19T03:24:34.827 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-19T03:24:34.829 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi066.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-19T03:24:35.589 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-19T03:24:37.944 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-19T03:24:41.028 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-19T03:24:41.054 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:41 +0000 (0:00:00.101) 0:00:00.101 ***** 2020-11-19T03:24:41.067 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:41 +0000 (0:00:00.013) 0:00:00.114 ***** 2020-11-19T03:24:42.789 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi066.front.sepia.ceph.com] 2020-11-19T03:24:42.879 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:42 +0000 (0:00:01.811) 0:00:01.926 ***** 2020-11-19T03:24:42.890 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] 2020-11-19T03:24:42.989 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:42 +0000 (0:00:00.110) 0:00:02.036 ***** 2020-11-19T03:24:43.150 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-19T03:24:43.185 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:43 +0000 (0:00:00.196) 0:00:02.232 ***** 2020-11-19T03:24:43.191 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-19T03:24:43.240 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:43 +0000 (0:00:00.053) 0:00:02.286 ***** 2020-11-19T03:24:43.636 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi066.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-19T03:24:43.673 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:43 +0000 (0:00:00.433) 0:00:02.720 ***** 2020-11-19T03:24:43.680 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-19T03:24:43.717 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:43 +0000 (0:00:00.043) 0:00:02.764 ***** 2020-11-19T03:24:43.993 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-19T03:24:43.994 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.070591", "end": "2020-11-19 03:24:43.977460", "rc": 0, "start": "2020-11-19 03:24:43.906869", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-19T03:24:44.031 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:44 +0000 (0:00:00.314) 0:00:03.078 ***** 2020-11-19T03:24:44.037 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.077567", "end": "2020-11-19 03:24:43.989728", "rc": 0, "start": "2020-11-19 03:24:43.912161", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-19T03:24:44.077 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:44 +0000 (0:00:00.046) 0:00:03.124 ***** 2020-11-19T03:24:44.286 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T03:24:44.322 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:44 +0000 (0:00:00.245) 0:00:03.369 ***** 2020-11-19T03:24:44.329 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T03:24:44.368 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:44 +0000 (0:00:00.045) 0:00:03.415 ***** 2020-11-19T03:24:45.060 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-19T03:24:45.062 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605756284.3353531-117157089051663/source", "state": "file", "uid": 0} 2020-11-19T03:24:45.097 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:45 +0000 (0:00:00.728) 0:00:04.144 ***** 2020-11-19T03:24:45.105 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605756284.3846269-100070186864463/source", "state": "file", "uid": 0} 2020-11-19T03:24:45.150 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:45 +0000 (0:00:00.049) 0:00:04.194 ***** 2020-11-19T03:24:45.403 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-19T03:24:45.405 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-19T03:24:45.406 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-19T03:24:45.450 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-19T03:24:45.452 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-19T03:24:45.598 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-19T03:24:45.604 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-19T03:24:45.753 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:45 +0000 (0:00:00.605) 0:00:04.800 ***** 2020-11-19T03:24:45.787 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:45 +0000 (0:00:00.034) 0:00:04.834 ***** 2020-11-19T03:24:46.539 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi154.front.sepia.ceph.com] 2020-11-19T03:24:46.549 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] 2020-11-19T03:24:46.655 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-19T03:24:46.656 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:46 +0000 (0:00:00.868) 0:00:05.702 ***** 2020-11-19T03:24:46.692 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:46 +0000 (0:00:00.037) 0:00:05.739 ***** 2020-11-19T03:24:47.132 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-19T03:24:47.134 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.231544", "end": "2020-11-19 03:24:47.108226", "rc": 0, "start": "2020-11-19 03:24:46.876682", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T03:24:47.180 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.487) 0:00:06.227 ***** 2020-11-19T03:24:47.187 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.254109", "end": "2020-11-19 03:24:47.115828", "rc": 0, "start": "2020-11-19 03:24:46.861719", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T03:24:47.226 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.046) 0:00:06.273 ***** 2020-11-19T03:24:47.361 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.015245", "end": "2020-11-19 03:24:47.327987", "failed_when_result": false, "rc": 0, "start": "2020-11-19 03:24:47.312742", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T03:24:47.410 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.178) 0:00:06.452 ***** 2020-11-19T03:24:47.412 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009567", "end": "2020-11-19 03:24:47.366948", "failed_when_result": false, "rc": 0, "start": "2020-11-19 03:24:47.357381", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-19T03:24:47.454 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.046) 0:00:06.498 ***** 2020-11-19T03:24:47.458 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-19T03:24:47.460 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-19T03:24:47.498 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.046) 0:00:06.545 ***** 2020-11-19T03:24:47.514 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-19T03:24:47.584 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.084) 0:00:06.629 ***** 2020-11-19T03:24:47.590 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.626 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.042) 0:00:06.672 ***** 2020-11-19T03:24:47.631 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.671 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.045) 0:00:06.718 ***** 2020-11-19T03:24:47.678 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.714 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.042) 0:00:06.761 ***** 2020-11-19T03:24:47.721 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.757 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.042) 0:00:06.804 ***** 2020-11-19T03:24:47.764 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-19T03:24:47.765 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.802 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.044) 0:00:06.849 ***** 2020-11-19T03:24:47.809 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.849 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.044) 0:00:06.893 ***** 2020-11-19T03:24:47.853 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-19T03:24:47.858 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.892 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.046) 0:00:06.939 ***** 2020-11-19T03:24:47.898 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.941 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.048) 0:00:06.988 ***** 2020-11-19T03:24:47.948 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-19T03:24:47.949 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:47.999 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:47 +0000 (0:00:00.055) 0:00:07.043 ***** 2020-11-19T03:24:48.004 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.045 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:48 +0000 (0:00:00.047) 0:00:07.090 ***** 2020-11-19T03:24:48.070 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-19T03:24:48.074 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.075 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.075 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.077 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.089 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.099 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.101 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.111 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.142 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.146 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f30 2020-11-19T03:24:48.155 INFO:teuthology.task.ansible.out:09adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n" 2020-11-19T03:24:48.161 INFO:teuthology.task.ansible.out:}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.167 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.169 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVN 2020-11-19T03:24:48.173 INFO:teuthology.task.ansible.out:TUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654 2020-11-19T03:24:48.178 INFO:teuthology.task.ansible.out:\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.181 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.184 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.186 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.197 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.202 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.208 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.210 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.227 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2020-11-19T03:24:48.230 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.234 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.238 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.249 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.252 INFO:teuthology.task.ansible.out: skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.258 INFO:teuthology.task.ansible.out: 2020-11-19T03:24:48.265 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw 2020-11-19T03:24:48.268 INFO:teuthology.task.ansible.out:ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.278 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9 2020-11-19T03:24:48.280 INFO:teuthology.task.ansible.out:aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.295 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"nam 2020-11-19T03:24:48.297 INFO:teuthology.task.ansible.out:e": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.304 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.316 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.346 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75 2020-11-19T03:24:48.351 INFO:teuthology.task.ansible.out:a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2 2020-11-19T03:24:48.358 INFO:teuthology.task.ansible.out:\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.374 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.382 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.395 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.413 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.417 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.430 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.444 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.455 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.466 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.468 INFO:teuthology.task.ansible.out: skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.480 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.498 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "chang 2020-11-19T03:24:48.502 INFO:teuthology.task.ansible.out:ed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.510 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.523 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.524 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.538 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.539 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.539 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.544 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, 2020-11-19T03:24:48.550 INFO:teuthology.task.ansible.out: "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.556 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4W 2020-11-19T03:24:48.560 INFO:teuthology.task.ansible.out:cNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.567 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.570 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.583 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.586 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.596 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.598 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.619 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.620 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.621 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.639 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.640 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.641 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.655 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.657 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.672 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eN 2020-11-19T03:24:48.678 INFO:teuthology.task.ansible.out:KOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.686 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.700 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.711 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.722 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.730 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.744 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.755 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.756 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.757 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.770 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.783 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.786 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.802 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.814 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.816 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.817 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.828 INFO:teuthology.task.ansible.out: skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2 2020-11-19T03:24:48.830 INFO:teuthology.task.ansible.out:xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.832 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.834 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.846 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.846 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.862 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.863 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.879 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.891 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.894 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.906 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.917 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.931 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.946 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.961 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.978 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:48.994 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.018 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": 2020-11-19T03:24:49.020 INFO:teuthology.task.ansible.out:"jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.038 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.040 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.056 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.061 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.070 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.074 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.089 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.099 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.106 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.115 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.126 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.128 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.137 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.152 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.153 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.165 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.193 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.194 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.195 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.195 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.196 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "ch 2020-11-19T03:24:49.198 INFO:teuthology.task.ansible.out:anged": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.250 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:01.206) 0:00:08.297 ***** 2020-11-19T03:24:49.261 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansib 2020-11-19T03:24:49.263 INFO:teuthology.task.ansible.out:le_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.272 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-19T03:24:49.274 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi066.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.326 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.075) 0:00:08.373 ***** 2020-11-19T03:24:49.361 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.034) 0:00:08.407 ***** 2020-11-19T03:24:49.367 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-19T03:24:49.406 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.045) 0:00:08.453 ***** 2020-11-19T03:24:49.413 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-19T03:24:49.415 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.481 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.075) 0:00:08.528 ***** 2020-11-19T03:24:49.492 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.549 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.067) 0:00:08.596 ***** 2020-11-19T03:24:49.559 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.600 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.050) 0:00:08.647 ***** 2020-11-19T03:24:49.606 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.647 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.047) 0:00:08.694 ***** 2020-11-19T03:24:49.655 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-19T03:24:49.658 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.700 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.053) 0:00:08.747 ***** 2020-11-19T03:24:49.711 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.754 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.054) 0:00:08.802 ***** 2020-11-19T03:24:49.761 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-19T03:24:49.763 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.800 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.044) 0:00:08.846 ***** 2020-11-19T03:24:49.805 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.842 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.043) 0:00:08.890 ***** 2020-11-19T03:24:49.849 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-11-19T03:24:49.851 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.887 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.044) 0:00:08.934 ***** 2020-11-19T03:24:49.894 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.937 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:49 +0000 (0:00:00.049) 0:00:08.984 ***** 2020-11-19T03:24:49.945 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-19T03:24:49.947 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.948 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.959 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.971 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.981 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:49.983 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.013 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.020 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.022 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.024 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.025 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.038 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.040 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.042 INFO:teuthology.task.ansible.out: skipping: [smithi066.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.044 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.046 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.054 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.060 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.074 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.092 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.110 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.126 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.130 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.137 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.157 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.170 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.178 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=opryp 2020-11-19T03:24:50.182 INFO:teuthology.task.ansible.out:in) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.189 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.193 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.201 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.203 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.218 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.234 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.239 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.240 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.244 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.322 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:50 +0000 (0:00:00.375) 0:00:09.360 ***** skipping: [smithi066.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.329 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:50.377 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:50 +0000 (0:00:00.062) 0:00:09.422 ***** 2020-11-19T03:24:50.990 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-19T03:24:51.029 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.652) 0:00:10.075 ***** 2020-11-19T03:24:51.063 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.033) 0:00:10.108 ***** 2020-11-19T03:24:51.070 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-19T03:24:51.074 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => { "msg": "Host smithi066.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-19T03:24:51.134 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.069) 0:00:10.178 ***** 2020-11-19T03:24:51.139 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => { "msg": "Host smithi154.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-19T03:24:51.185 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.051) 0:00:10.230 ***** 2020-11-19T03:24:51.190 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi066.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-19T03:24:51.228 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.045) 0:00:10.275 ***** 2020-11-19T03:24:51.253 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-19T03:24:51.286 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.057) 0:00:10.333 ***** 2020-11-19T03:24:51.293 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.342 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.046) 0:00:10.379 ***** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.404 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.064) 0:00:10.444 ***** 2020-11-19T03:24:51.407 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-19T03:24:51.409 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.465 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.067) 0:00:10.512 ***** 2020-11-19T03:24:51.477 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.517 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.052) 0:00:10.564 ***** 2020-11-19T03:24:51.523 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-19T03:24:51.525 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.569 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.047) 0:00:10.612 ***** 2020-11-19T03:24:51.593 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.646 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.079) 0:00:10.692 ***** 2020-11-19T03:24:51.659 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.711 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.066) 0:00:10.758 ***** 2020-11-19T03:24:51.721 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.776 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.063) 0:00:10.822 ***** 2020-11-19T03:24:51.784 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.835 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.057) 0:00:10.879 ***** 2020-11-19T03:24:51.841 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.890 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.057) 0:00:10.936 ***** 2020-11-19T03:24:51.896 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:51.945 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:51 +0000 (0:00:00.055) 0:00:10.992 ***** 2020-11-19T03:24:51.955 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.014 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.068) 0:00:11.061 ***** 2020-11-19T03:24:52.022 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-19T03:24:52.024 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.059 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.045) 0:00:11.106 ***** 2020-11-19T03:24:52.066 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.105 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.046) 0:00:11.152 ***** 2020-11-19T03:24:52.111 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-19T03:24:52.113 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.151 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.045) 0:00:11.198 ***** 2020-11-19T03:24:52.158 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.206 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.053) 0:00:11.252 ***** 2020-11-19T03:24:52.244 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi066.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi066.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:52.258 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.051) 0:00:11.303 ***** 2020-11-19T03:24:52.263 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:52.265 INFO:teuthology.task.ansible.out: skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:52.303 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.047) 0:00:11.350 ***** 2020-11-19T03:24:52.313 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-19T03:24:52.314 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.357 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.052) 0:00:11.403 ***** 2020-11-19T03:24:52.364 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.403 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.046) 0:00:11.449 ***** 2020-11-19T03:24:52.409 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-19T03:24:52.417 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.446 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.044) 0:00:11.493 ***** 2020-11-19T03:24:52.453 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.492 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.045) 0:00:11.539 ***** 2020-11-19T03:24:52.498 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-19T03:24:52.506 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:52.541 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.048) 0:00:11.588 ***** 2020-11-19T03:24:52.548 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:52.586 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.045) 0:00:11.633 ***** 2020-11-19T03:24:52.594 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-19T03:24:52.595 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.633 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.046) 0:00:11.680 ***** 2020-11-19T03:24:52.640 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.678 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.045) 0:00:11.725 ***** 2020-11-19T03:24:52.684 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-19T03:24:52.686 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:52.722 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.043) 0:00:11.769 ***** 2020-11-19T03:24:52.729 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:52.766 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.043) 0:00:11.813 ***** 2020-11-19T03:24:52.774 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.818 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.046) 0:00:11.859 ***** 2020-11-19T03:24:52.830 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.866 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.053) 0:00:11.913 ***** 2020-11-19T03:24:52.875 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.912 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.043) 0:00:11.956 ***** 2020-11-19T03:24:52.917 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:52.955 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.045) 0:00:12.001 ***** 2020-11-19T03:24:52.962 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.000 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:52 +0000 (0:00:00.044) 0:00:12.046 ***** 2020-11-19T03:24:53.006 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.064 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.064) 0:00:12.110 ***** 2020-11-19T03:24:53.075 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.109 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.046) 0:00:12.156 ***** 2020-11-19T03:24:53.116 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.157 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.047) 0:00:12.204 ***** 2020-11-19T03:24:53.163 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-19T03:24:53.176 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:53.203 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.045) 0:00:12.249 ***** 2020-11-19T03:24:53.213 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:53.260 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.048) 0:00:12.298 ***** TASK [common : Clean yum cache] ************************************************ skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.300 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.048) 0:00:12.347 ***** 2020-11-19T03:24:53.309 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.348 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.048) 0:00:12.395 ***** 2020-11-19T03:24:53.354 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-19T03:24:53.357 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.393 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.043) 0:00:12.438 ***** 2020-11-19T03:24:53.399 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.438 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.046) 0:00:12.485 ***** 2020-11-19T03:24:53.445 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-19T03:24:53.450 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.492 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.054) 0:00:12.539 ***** 2020-11-19T03:24:53.499 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.553 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.060) 0:00:12.600 ***** 2020-11-19T03:24:53.562 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.602 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.048) 0:00:12.649 ***** 2020-11-19T03:24:53.610 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.648 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.046) 0:00:12.695 ***** 2020-11-19T03:24:53.654 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-19T03:24:53.656 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.695 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.047) 0:00:12.742 ***** 2020-11-19T03:24:53.701 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.739 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.044) 0:00:12.786 ***** 2020-11-19T03:24:53.746 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.784 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.044) 0:00:12.831 ***** 2020-11-19T03:24:53.790 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.827 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.043) 0:00:12.874 ***** 2020-11-19T03:24:53.833 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-19T03:24:53.838 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:53.874 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.046) 0:00:12.921 ***** 2020-11-19T03:24:53.881 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:24:53.917 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.042) 0:00:12.964 ***** 2020-11-19T03:24:53.924 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-19T03:24:53.926 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:53.970 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:53 +0000 (0:00:00.052) 0:00:13.016 ***** 2020-11-19T03:24:53.976 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.022 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.052) 0:00:13.069 ***** 2020-11-19T03:24:54.030 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.077 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.054) 0:00:13.124 ***** 2020-11-19T03:24:54.083 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.130 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.052) 0:00:13.177 ***** 2020-11-19T03:24:54.137 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-19T03:24:54.138 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.184 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.054) 0:00:13.231 ***** 2020-11-19T03:24:54.192 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.239 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.054) 0:00:13.286 ***** 2020-11-19T03:24:54.245 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-19T03:24:54.247 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.289 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.050) 0:00:13.336 ***** 2020-11-19T03:24:54.296 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.298 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.339 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.049) 0:00:13.386 ***** 2020-11-19T03:24:54.348 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi066.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.390 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.051) 0:00:13.437 ***** 2020-11-19T03:24:54.397 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.398 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.437 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.046) 0:00:13.484 ***** 2020-11-19T03:24:54.443 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-19T03:24:54.445 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.503 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.065) 0:00:13.550 ***** 2020-11-19T03:24:54.509 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.548 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.045) 0:00:13.595 ***** 2020-11-19T03:24:54.644 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.007025", "end": "2020-11-19 03:24:54.617936", "rc": 0, "start": "2020-11-19 03:24:54.610911", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-19T03:24:54.685 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.136) 0:00:13.732 ***** 2020-11-19T03:24:54.712 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-19T03:24:54.714 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Get the current timezone] *************************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.025693", "end": "2020-11-19 03:24:54.692172", "rc": 0, "start": "2020-11-19 03:24:54.666479", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-19T03:24:54.753 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.067) 0:00:13.799 ***** 2020-11-19T03:24:54.789 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.036) 0:00:13.836 ***** 2020-11-19T03:24:54.795 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-19T03:24:54.797 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.835 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.046) 0:00:13.882 ***** 2020-11-19T03:24:54.843 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-19T03:24:54.845 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.880 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.044) 0:00:13.927 ***** 2020-11-19T03:24:54.886 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-19T03:24:54.888 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.925 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.045) 0:00:13.972 ***** 2020-11-19T03:24:54.932 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-19T03:24:54.935 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:54.972 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:54 +0000 (0:00:00.046) 0:00:14.019 ***** 2020-11-19T03:24:54.982 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-19T03:24:54.984 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.019 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.047) 0:00:14.066 ***** 2020-11-19T03:24:55.026 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-19T03:24:55.028 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.064 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.044) 0:00:14.111 ***** 2020-11-19T03:24:55.070 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-19T03:24:55.071 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.107 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.043) 0:00:14.154 ***** 2020-11-19T03:24:55.113 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-19T03:24:55.116 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.150 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.043) 0:00:14.197 ***** 2020-11-19T03:24:55.157 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-19T03:24:55.160 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.193 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.042) 0:00:14.240 ***** 2020-11-19T03:24:55.200 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-19T03:24:55.203 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.242 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.047) 0:00:14.288 ***** 2020-11-19T03:24:55.247 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-19T03:24:55.249 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.286 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.045) 0:00:14.334 ***** 2020-11-19T03:24:55.293 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-19T03:24:55.295 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.338 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.051) 0:00:14.385 ***** 2020-11-19T03:24:55.367 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.404 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.065) 0:00:14.451 ***** 2020-11-19T03:24:55.522 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.081394", "end": "2020-11-19 03:24:55.500372", "rc": 0, "start": "2020-11-19 03:24:55.418978", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:24:55.570 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.158) 0:00:14.610 ***** 2020-11-19T03:24:55.593 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:55.632 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.068) 0:00:14.679 ***** 2020-11-19T03:24:55.799 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023856", "end": "2020-11-19 03:24:55.770529", "rc": 0, "start": "2020-11-19 03:24:55.746673", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:24:55.837 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:55 +0000 (0:00:00.205) 0:00:14.884 ***** 2020-11-19T03:24:56.530 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.987857", "end": "2020-11-19 03:24:56.500369", "rc": 0, "start": "2020-11-19 03:24:55.512512", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:24:56.573 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:56 +0000 (0:00:00.729) 0:00:15.614 ***** 2020-11-19T03:24:56.595 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-19T03:24:56.597 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:24:56.631 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:56 +0000 (0:00:00.064) 0:00:15.678 ***** 2020-11-19T03:24:56.792 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-19T03:24:57.011 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.031695", "end": "2020-11-19 03:24:56.768511", "rc": 0, "start": "2020-11-19 03:24:56.736816", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 19 November 2020 03:24:56 +0000 (0:00:00.205) 0:00:15.883 ***** 2020-11-19T03:24:59.684 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi066.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:24:59.722 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:24:59 +0000 (0:00:02.884) 0:00:18.768 ***** 2020-11-19T03:25:00.249 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:00.290 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:00 +0000 (0:00:00.566) 0:00:19.335 ***** 2020-11-19T03:25:00.607 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:00.643 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:00 +0000 (0:00:00.355) 0:00:19.690 ***** 2020-11-19T03:25:00.670 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:00.706 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:00 +0000 (0:00:00.062) 0:00:19.753 ***** 2020-11-19T03:25:01.051 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-19T03:25:01.056 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-19T03:25:01.090 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:01 +0000 (0:00:00.384) 0:00:20.137 ***** 2020-11-19T03:25:01.154 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:01.191 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:01 +0000 (0:00:00.100) 0:00:20.238 ***** 2020-11-19T03:25:01.220 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:01.257 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:01 +0000 (0:00:00.066) 0:00:20.305 ***** 2020-11-19T03:25:01.588 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-19T03:25:01.590 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-19T03:25:01.630 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:01 +0000 (0:00:00.367) 0:00:20.672 ***** 2020-11-19T03:25:01.981 INFO:teuthology.task.ansible.out: 2020-11-19T03:25:01.986 INFO:teuthology.task.ansible.out:TASK [common : Make sure smartmontools is installed] *************************** 2020-11-19T03:25:01.989 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:02.037 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:02 +0000 (0:00:00.411) 0:00:21.084 ***** 2020-11-19T03:25:02.381 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-19T03:25:02.382 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-19T03:25:02.550 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:02.585 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:02 +0000 (0:00:00.547) 0:00:21.632 ***** 2020-11-19T03:25:02.711 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi066.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-19T03:25:03.026 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-19T03:25:03.069 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:03 +0000 (0:00:00.483) 0:00:22.115 ***** 2020-11-19T03:25:03.081 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-19T03:25:03.196 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-19T03:25:03.198 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T03:25:03.249 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:03 +0000 (0:00:00.180) 0:00:22.296 ***** 2020-11-19T03:25:03.543 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-19T03:25:03.545 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-19T03:25:03.612 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-19T03:25:03.613 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-19T03:25:03.957 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-19T03:25:04.039 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-19T03:25:04.045 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-19T03:25:04.104 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.855) 0:00:23.151 ***** 2020-11-19T03:25:04.237 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T03:25:04.278 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.171) 0:00:23.323 ***** 2020-11-19T03:25:04.304 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi066.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-19T03:25:04.351 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.075) 0:00:23.398 ***** 2020-11-19T03:25:04.392 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-19T03:25:04.394 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-19T03:25:04.432 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.080) 0:00:23.479 ***** 2020-11-19T03:25:04.461 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:04.499 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.067) 0:00:23.546 ***** 2020-11-19T03:25:04.526 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi066.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:25:04.564 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.064) 0:00:23.611 ***** 2020-11-19T03:25:04.601 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-19T03:25:04.603 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-19T03:25:04.612 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-19T03:25:04.615 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:04.626 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:04.663 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.098) 0:00:23.710 ***** 2020-11-19T03:25:04.690 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-19T03:25:04.692 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:04.741 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:04 +0000 (0:00:00.076) 0:00:23.786 ***** 2020-11-19T03:25:04.865 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-19T03:25:04.869 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-19T03:25:05.155 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-19T03:25:05.201 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:05 +0000 (0:00:00.461) 0:00:24.248 ***** 2020-11-19T03:25:05.241 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-19T03:25:05.277 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:05 +0000 (0:00:00.075) 0:00:24.324 ***** 2020-11-19T03:25:05.304 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:05.340 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:05 +0000 (0:00:00.063) 0:00:24.387 ***** 2020-11-19T03:25:05.368 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:25:05.403 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:05 +0000 (0:00:00.062) 0:00:24.450 ***** 2020-11-19T03:25:05.442 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:05.463 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:05.477 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:05.538 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:05 +0000 (0:00:00.134) 0:00:24.584 ***** 2020-11-19T03:25:05.565 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:05.604 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:05 +0000 (0:00:00.067) 0:00:24.652 ***** 2020-11-19T03:25:05.673 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-19T03:25:05.675 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:05.712 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:05 +0000 (0:00:00.107) 0:00:24.759 ***** 2020-11-19T03:25:06.086 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-19T03:25:06.125 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:06 +0000 (0:00:00.412) 0:00:25.172 ***** 2020-11-19T03:25:06.484 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-19T03:25:06.486 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-19T03:25:06.521 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:06 +0000 (0:00:00.395) 0:00:25.567 ***** 2020-11-19T03:25:06.528 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-19T03:25:06.530 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:06.566 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:06 +0000 (0:00:00.044) 0:00:25.612 ***** 2020-11-19T03:25:06.684 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-19T03:25:06.695 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-19T03:25:06.890 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-19T03:25:06.940 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:06 +0000 (0:00:00.374) 0:00:25.987 ***** 2020-11-19T03:25:06.947 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-19T03:25:06.949 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-19T03:25:06.989 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:06 +0000 (0:00:00.047) 0:00:26.034 ***** 2020-11-19T03:25:06.995 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-19T03:25:06.997 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:07.057 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.068) 0:00:26.103 ***** 2020-11-19T03:25:07.100 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-19T03:25:07.102 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:25:07.139 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.083) 0:00:26.187 ***** 2020-11-19T03:25:07.167 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-19T03:25:07.169 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:07.203 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.063) 0:00:26.250 ***** 2020-11-19T03:25:07.338 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-19T03:25:07.377 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.173) 0:00:26.424 ***** 2020-11-19T03:25:07.401 INFO:teuthology.task.ansible.out: 2020-11-19T03:25:07.574 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-19T03:25:07.576 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-19T03:25:07.613 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.236) 0:00:26.660 ***** 2020-11-19T03:25:07.687 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi154.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-19T03:25:07.733 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.120) 0:00:26.780 ***** 2020-11-19T03:25:07.772 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:07.814 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.078) 0:00:26.859 ***** 2020-11-19T03:25:07.846 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi154.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-19T03:25:07.894 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.077) 0:00:26.937 ***** 2020-11-19T03:25:07.936 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:07.974 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:07 +0000 (0:00:00.082) 0:00:27.019 ***** 2020-11-19T03:25:08.246 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 03:24:15 UTC", "ActiveEnterTimestampMonotonic": "21205230", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount var-run.mount nss-lookup.target systemd-journald.socket basic.target sysinit.target time-sync.target systemd-tmpfiles-setup.service network.target local-fs.target remote-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 03:24:15 UTC", "AssertTimestampMonotonic": "21203354", "Before": "shutdown.target getty@tty1.service multi-user.target xdm.service plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 03:24:15 UTC", "ConditionTimestampMonotonic": "21203354", "ConfigurationDirectoryMode": "0755", "Confl 2020-11-19T03:25:08.250 INFO:teuthology.task.ansible.out:icts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1154", "ExecMainStartTimestamp": "Thu 2020-11-19 03:24:15 UTC", "ExecMainStartTimestampMonotonic": "21205147", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-11-19 03:24:15 UTC] ; stop_time=[n/a] ; pid=1154 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 03:24:15 UTC", "InactiveExitTimestampMonotonic": "21205230", "InvocationID": "b3f6740ff3be4d8e8bfa4e5e34bd185b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "Key 2020-11-19T03:25:08.254 INFO:teuthology.task.ansible.out:ringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1154", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", 2020-11-19T03:25:08.260 INFO:teuthology.task.ansible.out:"RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 03:24:15 UTC", "StateChangeTimestampMonotonic": "21205230", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 03:24:15 UTC", "WatchdogTimestampMonotonic": "21205228", "WatchdogUSec": "0"}} 2020-11-19T03:25:08.284 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.310) 0:00:27.330 ***** 2020-11-19T03:25:08.318 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.353 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.070) 0:00:27.400 ***** 2020-11-19T03:25:08.360 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-19T03:25:08.362 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-19T03:25:08.406 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.053) 0:00:27.453 ***** 2020-11-19T03:25:08.413 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-19T03:25:08.415 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.453 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.046) 0:00:27.500 ***** 2020-11-19T03:25:08.454 INFO:teuthology.task.ansible.out: 2020-11-19T03:25:08.494 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-19T03:25:08.497 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.531 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.078) 0:00:27.578 ***** 2020-11-19T03:25:08.561 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.596 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.064) 0:00:27.643 ***** 2020-11-19T03:25:08.645 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-19T03:25:08.646 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.682 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.084) 0:00:27.727 ***** 2020-11-19T03:25:08.688 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-19T03:25:08.693 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 03:24:27 UTC", "ActiveEnterTimestampMonotonic": "21418960", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target system.slice time-sync.target remote-fs.target local-fs.target basic.target systemd-journald.socket -.mount var-run.mount systemd-tmpfiles-setup.service sysinit.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 03:24:27 UTC", "AssertTimestampMonotonic": "21417440", "Before": "xdm.service shutdown.target plymouth-quit.service getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 03:24:27 UTC", "ConditionTimestampMonotonic": "21417440", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-19T03:25:08.698 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1295", "ExecMainStartTimestamp": "Thu 2020-11-19 03:24:27 UTC", "ExecMainStartTimestampMonotonic": "21418911", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-11-19 03:24:27 UTC] ; stop_time=[n/a] ; pid=1295 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 03:24:27 UTC", "InactiveExitTimestampMonotonic": "21418960", "InvocationID": "064b44a68f8345079c953b0e7566c683", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-11-19T03:25:08.702 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1295", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-11-19T03:25:08.705 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 03:24:27 UTC", "StateChangeTimestampMonotonic": "21418960", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 03:24:27 UTC", "WatchdogTimestampMonotonic": "21418959", "WatchdogUSec": "0"}} 2020-11-19T03:25:08.730 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.050) 0:00:27.778 ***** 2020-11-19T03:25:08.741 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.776 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.045) 0:00:27.823 ***** 2020-11-19T03:25:08.786 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.825 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.049) 0:00:27.872 ***** 2020-11-19T03:25:08.832 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-19T03:25:08.834 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.885 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.051) 0:00:27.924 ***** TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.926 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.047) 0:00:27.972 ***** 2020-11-19T03:25:08.933 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:08.974 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:08 +0000 (0:00:00.048) 0:00:28.020 ***** 2020-11-19T03:25:08.980 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-19T03:25:08.982 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.017 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.043) 0:00:28.064 ***** 2020-11-19T03:25:09.024 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.063 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.045) 0:00:28.110 ***** 2020-11-19T03:25:09.069 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-19T03:25:09.071 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.113 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.046) 0:00:28.156 ***** 2020-11-19T03:25:09.115 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-19T03:25:09.116 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.154 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.044) 0:00:28.201 ***** 2020-11-19T03:25:09.160 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-19T03:25:09.162 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.197 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.043) 0:00:28.244 ***** 2020-11-19T03:25:09.204 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-19T03:25:09.240 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.043) 0:00:28.287 ***** 2020-11-19T03:25:09.247 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-19T03:25:09.249 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.285 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.044) 0:00:28.332 ***** 2020-11-19T03:25:09.292 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-19T03:25:09.294 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-19T03:25:09.330 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.045) 0:00:28.377 ***** 2020-11-19T03:25:09.336 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-19T03:25:09.339 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.375 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.044) 0:00:28.421 ***** 2020-11-19T03:25:09.381 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-19T03:25:09.382 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-19T03:25:09.419 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.044) 0:00:28.466 ***** 2020-11-19T03:25:09.425 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-19T03:25:09.428 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.463 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.043) 0:00:28.510 ***** 2020-11-19T03:25:09.468 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-19T03:25:09.470 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-19T03:25:09.509 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.046) 0:00:28.556 ***** 2020-11-19T03:25:09.516 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-19T03:25:09.518 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.554 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.044) 0:00:28.601 ***** 2020-11-19T03:25:09.606 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.645 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.091) 0:00:28.692 ***** 2020-11-19T03:25:09.651 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-19T03:25:09.653 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-19T03:25:09.697 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.051) 0:00:28.744 ***** 2020-11-19T03:25:09.705 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-19T03:25:09.741 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.044) 0:00:28.788 ***** 2020-11-19T03:25:09.779 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-19T03:25:09.781 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-19T03:25:09.816 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.075) 0:00:28.864 ***** 2020-11-19T03:25:09.833 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-19T03:25:09.835 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-19T03:25:09.871 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.054) 0:00:28.918 ***** 2020-11-19T03:25:09.878 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-19T03:25:09.880 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-19T03:25:09.928 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.054) 0:00:28.973 ***** 2020-11-19T03:25:09.935 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:09.992 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:09 +0000 (0:00:00.065) 0:00:29.039 ***** 2020-11-19T03:25:10.000 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi154.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-19T03:25:10.061 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.068) 0:00:29.107 ***** 2020-11-19T03:25:10.075 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-19T03:25:10.078 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:10.116 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.056) 0:00:29.163 ***** 2020-11-19T03:25:10.199 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-19T03:25:10.201 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-19T03:25:10.240 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.122) 0:00:29.286 ***** 2020-11-19T03:25:10.332 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-19T03:25:10.333 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-19T03:25:10.369 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.130) 0:00:29.416 ***** 2020-11-19T03:25:10.386 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-19T03:25:10.388 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-19T03:25:10.423 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.053) 0:00:29.470 ***** 2020-11-19T03:25:10.449 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-19T03:25:10.451 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:10.485 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.061) 0:00:29.532 ***** 2020-11-19T03:25:10.513 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-19T03:25:10.514 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:10.552 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.066) 0:00:29.599 ***** 2020-11-19T03:25:10.706 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-19T03:25:10.708 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-19T03:25:10.743 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.191) 0:00:29.790 ***** 2020-11-19T03:25:10.771 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-19T03:25:10.773 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-19T03:25:10.808 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.064) 0:00:29.855 ***** 2020-11-19T03:25:10.838 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-19T03:25:10.841 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi066"}, "changed": false} 2020-11-19T03:25:10.881 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:10 +0000 (0:00:00.072) 0:00:29.928 ***** 2020-11-19T03:25:11.173 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-19T03:25:11.208 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:11 +0000 (0:00:00.326) 0:00:30.255 ***** 2020-11-19T03:25:11.235 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-19T03:25:11.237 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi154"}, "changed": false} 2020-11-19T03:25:11.271 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:11 +0000 (0:00:00.063) 0:00:30.319 ***** 2020-11-19T03:25:11.879 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi154.front.sepia.ceph.com", "ansible_hostname": "smithi154", "ansible_nodename": "smithi154"}, "changed": true, "name": "smithi154"} 2020-11-19T03:25:11.932 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:11 +0000 (0:00:00.658) 0:00:30.977 ***** 2020-11-19T03:25:11.940 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi066.front.sepia.ceph.com", "ansible_hostname": "smithi066", "ansible_nodename": "smithi066"}, "changed": true, "name": "smithi066"} 2020-11-19T03:25:11.977 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:11 +0000 (0:00:00.047) 0:00:31.024 ***** 2020-11-19T03:25:12.263 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-19T03:25:12.310 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:12 +0000 (0:00:00.332) 0:00:31.357 ***** 2020-11-19T03:25:12.317 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-19T03:25:12.362 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:12 +0000 (0:00:00.052) 0:00:31.409 ***** 2020-11-19T03:25:12.610 INFO:teuthology.task.ansible.out: 2020-11-19T03:25:12.770 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-19T03:25:12.808 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:12 +0000 (0:00:00.445) 0:00:31.855 ***** 2020-11-19T03:25:12.858 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-19T03:25:12.898 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:12 +0000 (0:00:00.088) 0:00:31.944 ***** 2020-11-19T03:25:13.350 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-19T03:25:13.352 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-19T03:25:13.387 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.490) 0:00:32.434 ***** 2020-11-19T03:25:13.406 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-19T03:25:13.442 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.052) 0:00:32.487 ***** 2020-11-19T03:25:13.446 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-19T03:25:13.448 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.486 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.045) 0:00:32.533 ***** 2020-11-19T03:25:13.493 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.533 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.047) 0:00:32.580 ***** 2020-11-19T03:25:13.540 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.587 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.047) 0:00:32.628 ***** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.640 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.058) 0:00:32.687 ***** 2020-11-19T03:25:13.649 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.695 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.053) 0:00:32.740 ***** 2020-11-19T03:25:13.700 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.759 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.064) 0:00:32.805 ***** 2020-11-19T03:25:13.769 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.811 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.053) 0:00:32.858 ***** 2020-11-19T03:25:13.819 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.869 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.055) 0:00:32.913 ***** 2020-11-19T03:25:13.880 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.936 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.062) 0:00:32.975 ***** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:13.977 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:13 +0000 (0:00:00.048) 0:00:33.024 ***** 2020-11-19T03:25:13.984 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-19T03:25:13.986 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.033 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.054) 0:00:33.078 ***** 2020-11-19T03:25:14.038 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.086 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.054) 0:00:33.133 ***** 2020-11-19T03:25:14.097 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.130 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.044) 0:00:33.177 ***** 2020-11-19T03:25:14.137 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.175 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.044) 0:00:33.222 ***** 2020-11-19T03:25:14.182 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.227 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.052) 0:00:33.274 ***** 2020-11-19T03:25:14.235 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.275 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.048) 0:00:33.322 ***** 2020-11-19T03:25:14.283 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-19T03:25:14.318 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.042) 0:00:33.365 ***** 2020-11-19T03:25:14.362 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.043) 0:00:33.408 ***** 2020-11-19T03:25:14.368 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-19T03:25:14.410 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.048) 0:00:33.457 ***** 2020-11-19T03:25:14.463 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.052) 0:00:33.510 ***** 2020-11-19T03:25:14.471 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-19T03:25:14.507 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.044) 0:00:33.554 ***** 2020-11-19T03:25:14.550 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.042) 0:00:33.597 ***** 2020-11-19T03:25:14.556 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-19T03:25:14.558 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.607 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.057) 0:00:33.654 ***** 2020-11-19T03:25:14.614 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.658 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.050) 0:00:33.705 ***** 2020-11-19T03:25:14.667 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-19T03:25:14.671 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.714 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.056) 0:00:33.761 ***** 2020-11-19T03:25:14.721 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.721 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.760 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.045) 0:00:33.807 ***** 2020-11-19T03:25:14.767 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-19T03:25:14.769 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.804 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.044) 0:00:33.851 ***** 2020-11-19T03:25:14.810 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.852 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.047) 0:00:33.899 ***** 2020-11-19T03:25:14.861 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:25:14.863 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.907 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.055) 0:00:33.954 ***** 2020-11-19T03:25:14.914 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.951 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.044) 0:00:33.998 ***** 2020-11-19T03:25:14.958 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-19T03:25:14.959 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:14.994 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:14 +0000 (0:00:00.043) 0:00:34.042 ***** 2020-11-19T03:25:15.001 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.038 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.042) 0:00:34.085 ***** 2020-11-19T03:25:15.045 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.081 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.043) 0:00:34.128 ***** 2020-11-19T03:25:15.090 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.127 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.045) 0:00:34.174 ***** 2020-11-19T03:25:15.136 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.185 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.055) 0:00:34.230 ***** 2020-11-19T03:25:15.199 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.241 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.057) 0:00:34.288 ***** 2020-11-19T03:25:15.251 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.289 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.047) 0:00:34.336 ***** 2020-11-19T03:25:15.298 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.345 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.055) 0:00:34.391 ***** 2020-11-19T03:25:15.351 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.390 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.045) 0:00:34.437 ***** 2020-11-19T03:25:15.394 INFO:teuthology.task.ansible.out: 2020-11-19T03:25:15.398 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.449 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.045) 0:00:34.482 ***** TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.486 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.043) 0:00:34.526 ***** 2020-11-19T03:25:15.492 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.538 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.053) 0:00:34.580 ***** 2020-11-19T03:25:15.540 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.582 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.048) 0:00:34.628 ***** 2020-11-19T03:25:15.588 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.628 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.047) 0:00:34.675 ***** 2020-11-19T03:25:15.636 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.680 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.046) 0:00:34.722 ***** 2020-11-19T03:25:15.681 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.723 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.045) 0:00:34.768 ***** 2020-11-19T03:25:15.727 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-19T03:25:15.729 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.780 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.059) 0:00:34.827 ***** 2020-11-19T03:25:15.788 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.830 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.047) 0:00:34.875 ***** 2020-11-19T03:25:15.835 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.878 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.050) 0:00:34.925 ***** 2020-11-19T03:25:15.885 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.925 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.046) 0:00:34.972 ***** 2020-11-19T03:25:15.938 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:15.975 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:15 +0000 (0:00:00.050) 0:00:35.022 ***** 2020-11-19T03:25:15.982 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.022 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.046) 0:00:35.069 ***** 2020-11-19T03:25:16.031 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.068 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.046) 0:00:35.115 ***** 2020-11-19T03:25:16.076 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.126 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.052) 0:00:35.167 ***** 2020-11-19T03:25:16.134 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.172 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.049) 0:00:35.217 ***** 2020-11-19T03:25:16.176 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.214 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.043) 0:00:35.260 ***** 2020-11-19T03:25:16.222 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.262 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.046) 0:00:35.307 ***** 2020-11-19T03:25:16.270 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.311 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.050) 0:00:35.358 ***** 2020-11-19T03:25:16.318 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-19T03:25:16.321 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.360 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.048) 0:00:35.406 ***** 2020-11-19T03:25:16.367 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.406 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.045) 0:00:35.451 ***** 2020-11-19T03:25:16.411 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-19T03:25:16.413 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.451 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.046) 0:00:35.498 ***** 2020-11-19T03:25:16.457 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.496 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.045) 0:00:35.543 ***** 2020-11-19T03:25:16.503 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.545 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.048) 0:00:35.592 ***** 2020-11-19T03:25:16.551 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.590 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.045) 0:00:35.637 ***** 2020-11-19T03:25:16.602 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.640 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.050) 0:00:35.687 ***** 2020-11-19T03:25:16.646 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:16.687 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.046) 0:00:35.733 ***** 2020-11-19T03:25:16.790 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005203", "end": "2020-11-19 03:25:16.764901", "rc": 0, "start": "2020-11-19 03:25:16.759698", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:25:16.850 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.161) 0:00:35.895 ***** 2020-11-19T03:25:16.855 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004852", "end": "2020-11-19 03:25:16.816206", "rc": 0, "start": "2020-11-19 03:25:16.811354", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:25:16.905 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:16 +0000 (0:00:00.057) 0:00:35.953 ***** 2020-11-19T03:25:17.312 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** Thursday 19 November 2020 03:25:16 +0000 (0:00:00.044) 0:00:35.997 ***** Thursday 19 November 2020 03:25:17 +0000 (0:00:00.065) 0:00:36.063 ***** 2020-11-19T03:25:17.365 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-19T03:25:17.367 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-19T03:25:17.406 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:17 +0000 (0:00:00.390) 0:00:36.453 ***** 2020-11-19T03:25:17.436 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set apt preferences] ****************************************** 2020-11-19T03:25:17.438 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-19T03:25:17.477 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:17 +0000 (0:00:00.071) 0:00:36.525 ***** 2020-11-19T03:25:17.515 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:17 +0000 (0:00:00.037) 0:00:36.562 ***** 2020-11-19T03:25:17.522 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-19T03:25:17.523 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:17.565 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:17 +0000 (0:00:00.050) 0:00:36.612 ***** 2020-11-19T03:25:18.977 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi154.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-19T03:25:19.038 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-19T03:25:20.040 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-19T03:25:20.080 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:20 +0000 (0:00:02.514) 0:00:39.127 ***** 2020-11-19T03:25:20.392 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-19T03:25:20.447 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:20 +0000 (0:00:00.365) 0:00:39.493 ***** 2020-11-19T03:25:20.988 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-19T03:25:20.994 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:21.028 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:21 +0000 (0:00:00.582) 0:00:40.075 ***** 2020-11-19T03:25:21.058 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-19T03:25:21.093 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:21 +0000 (0:00:00.065) 0:00:40.140 ***** 2020-11-19T03:25:21.283 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo 2020-11-19T03:25:21.286 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034167", "end": "2020-11-19 03:25:21.249622", "rc": 0, "start": "2020-11-19 03:25:21.215455", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:25:21.323 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:21 +0000 (0:00:00.229) 0:00:40.370 ***** 2020-11-19T03:25:21.427 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-19T03:25:21.429 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:21.466 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:21 +0000 (0:00:00.143) 0:00:40.514 ***** 2020-11-19T03:25:21.500 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-19T03:25:21.540 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:21 +0000 (0:00:00.073) 0:00:40.587 ***** 2020-11-19T03:25:21.718 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034913", "end": "2020-11-19 03:25:21.690187", "rc": 0, "start": "2020-11-19 03:25:21.655274", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:25:21.755 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:21 +0000 (0:00:00.215) 0:00:40.802 ***** 2020-11-19T03:25:24.211 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-19T03:25:24.213 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:24.251 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:24 +0000 (0:00:02.495) 0:00:43.298 ***** 2020-11-19T03:25:24.633 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:24.670 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:24 +0000 (0:00:00.419) 0:00:43.717 ***** 2020-11-19T03:25:33.439 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-19T03:25:33.441 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137606 files and directorie 2020-11-19T03:25:33.444 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-19T03:25:33.448 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137606 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T03:25:33.477 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:33 +0000 (0:00:08.806) 0:00:52.524 ***** 2020-11-19T03:25:33.926 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137606 files and directorie 2020-11-19T03:25:33.928 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 13 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-19T03:25:33.930 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137606 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T03:25:33.964 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:33 +0000 (0:00:00.486) 0:00:53.011 ***** 2020-11-19T03:25:34.391 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:34.429 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:34 +0000 (0:00:00.465) 0:00:53.476 ***** 2020-11-19T03:25:34.433 INFO:teuthology.task.ansible.out: 2020-11-19T03:25:34.460 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:34.521 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:34 +0000 (0:00:00.088) 0:00:53.565 ***** 2020-11-19T03:25:34.880 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-19T03:25:34.882 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:34.922 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:34 +0000 (0:00:00.401) 0:00:53.967 ***** 2020-11-19T03:25:34.947 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:34.985 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:34 +0000 (0:00:00.065) 0:00:54.032 ***** 2020-11-19T03:25:43.394 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9641 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database 2020-11-19T03:25:43.396 INFO:teuthology.task.ansible.out: ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137464 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing trigger 2020-11-19T03:25:43.398 INFO:teuthology.task.ansible.out:s for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9641 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Read 2020-11-19T03:25:43.400 INFO:teuthology.task.ansible.out:ing database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137464 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T03:25:43.433 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:43 +0000 (0:00:08.448) 0:01:02.480 ***** 2020-11-19T03:25:44.072 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9540 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-11-19T03:25:44.074 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137464 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-11-19T03:25:44.076 INFO:teuthology.task.ansible.out:u1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9540 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-11-19T03:25:44.078 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137464 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-19T03:25:44.125 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:44 +0000 (0:00:00.692) 0:01:03.172 ***** 2020-11-19T03:25:44.367 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-19T03:25:44.369 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:44.414 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:44 +0000 (0:00:00.288) 0:01:03.461 ***** 2020-11-19T03:25:44.546 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-19T03:25:44.548 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-19T03:25:44.588 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:44 +0000 (0:00:00.173) 0:01:03.634 ***** 2020-11-19T03:25:44.614 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-19T03:25:44.617 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:44.655 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:44 +0000 (0:00:00.067) 0:01:03.702 ***** 2020-11-19T03:25:44.789 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-19T03:25:44.791 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-19T03:25:44.826 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:44 +0000 (0:00:00.171) 0:01:03.873 ***** 2020-11-19T03:25:45.034 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:45.077 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.249) 0:01:04.123 ***** 2020-11-19T03:25:45.135 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-19T03:25:45.138 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-19T03:25:45.181 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.105) 0:01:04.228 ***** 2020-11-19T03:25:45.210 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-19T03:25:45.254 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.072) 0:01:04.301 ***** 2020-11-19T03:25:45.296 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-19T03:25:45.297 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:45.336 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.082) 0:01:04.383 ***** 2020-11-19T03:25:45.353 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-19T03:25:45.355 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-19T03:25:45.416 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.079) 0:01:04.463 ***** 2020-11-19T03:25:45.475 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-19T03:25:45.477 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-19T03:25:45.516 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.100) 0:01:04.563 ***** 2020-11-19T03:25:45.544 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-19T03:25:45.546 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T03:25:45.597 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.080) 0:01:04.644 ***** 2020-11-19T03:25:45.852 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-19T03:25:45.854 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-19T03:25:45.893 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.295) 0:01:04.940 ***** 2020-11-19T03:25:45.900 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-19T03:25:45.902 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 03:24:27 UTC", "ActiveEnterTimestampMonotonic": "21733231", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target sysinit.target network.target systemd-tmpfiles-setup.service nss-lookup.target systemd-journald.socket -.mount remote-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 03:24:27 UTC", "AssertTimestampMonotonic": "21412392", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 03:24:27 UTC", "ConditionTimestampMonotonic": "21412392", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-19T03:25:45.903 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1361", "ExecMainStartTimestamp": "Thu 2020-11-19 03:24:27 UTC", "ExecMainStartTimestampMonotonic": "21733188", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 03:24:27 UTC", "InactiveExitTimestampMonotonic": "21414792", "InvocationID": "9e8fe3f182f545a8b9c2d6ce03df46ea", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-19T03:25:45.905 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1361", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-19T03:25:45.906 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 03:24:27 UTC", "StateChangeTimestampMonotonic": "21733231", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 03:24:27 UTC", "WatchdogTimestampMonotonic": "21733229", "WatchdogUSec": "0"}} 2020-11-19T03:25:45.944 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:45 +0000 (0:00:00.051) 0:01:04.991 ***** 2020-11-19T03:25:45.972 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-19T03:25:45.974 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.013 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.068) 0:01:05.059 ***** 2020-11-19T03:25:46.039 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-19T03:25:46.041 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.075 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.062) 0:01:05.122 ***** 2020-11-19T03:25:46.082 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-19T03:25:46.083 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-19T03:25:46.124 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.048) 0:01:05.171 ***** 2020-11-19T03:25:46.131 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-19T03:25:46.133 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.173 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.048) 0:01:05.220 ***** 2020-11-19T03:25:46.200 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-19T03:25:46.202 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.246 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.072) 0:01:05.293 ***** 2020-11-19T03:25:46.263 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-19T03:25:46.265 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T03:25:46.300 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.054) 0:01:05.347 ***** 2020-11-19T03:25:46.313 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.345 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.045) 0:01:05.393 ***** 2020-11-19T03:25:46.372 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-19T03:25:46.379 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.416 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.070) 0:01:05.463 ***** 2020-11-19T03:25:46.443 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-19T03:25:46.445 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.479 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.062) 0:01:05.526 ***** 2020-11-19T03:25:46.509 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.544 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.065) 0:01:05.591 ***** 2020-11-19T03:25:46.572 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:25:46.574 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.608 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.064) 0:01:05.655 ***** 2020-11-19T03:25:46.636 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-19T03:25:46.638 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.673 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.065) 0:01:05.720 ***** 2020-11-19T03:25:46.680 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-19T03:25:46.682 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 03:24:16 UTC", "ActiveEnterTimestampMonotonic": "21741333", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice -.mount systemd-journald.socket network.target systemd-tmpfiles-setup.service nss-lookup.target sysinit.target basic.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 03:24:15 UTC", "AssertTimestampMonotonic": "21201448", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 03:24:15 UTC", "ConditionTimestampMonotonic": "21201448", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-19T03:25:46.684 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1268", "ExecMainStartTimestamp": "Thu 2020-11-19 03:24:16 UTC", "ExecMainStartTimestampMonotonic": "21741307", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 03:24:15 UTC", "InactiveExitTimestampMonotonic": "21203264", "InvocationID": "e9698307634f4b2d9bb9e0bd3cf59242", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-19T03:25:46.687 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1268", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-19T03:25:46.689 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 03:24:16 UTC", "StateChangeTimestampMonotonic": "21741333", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 03:24:16 UTC", "WatchdogTimestampMonotonic": "21741331", "WatchdogUSec": "0"}} 2020-11-19T03:25:46.719 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.045) 0:01:05.766 ***** 2020-11-19T03:25:46.727 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-19T03:25:46.729 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.763 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.043) 0:01:05.810 ***** 2020-11-19T03:25:46.769 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-19T03:25:46.770 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.806 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.042) 0:01:05.853 ***** 2020-11-19T03:25:46.812 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-19T03:25:46.814 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.849 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.043) 0:01:05.897 ***** 2020-11-19T03:25:46.856 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-19T03:25:46.857 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.893 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.042) 0:01:05.940 ***** 2020-11-19T03:25:46.899 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-19T03:25:46.901 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.935 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.042) 0:01:05.982 ***** 2020-11-19T03:25:46.942 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-19T03:25:46.943 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:46.978 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:46 +0000 (0:00:00.042) 0:01:06.025 ***** 2020-11-19T03:25:46.985 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-19T03:25:46.986 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.021 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.042) 0:01:06.068 ***** 2020-11-19T03:25:47.034 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-19T03:25:47.035 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.069 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.048) 0:01:06.116 ***** 2020-11-19T03:25:47.076 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-19T03:25:47.078 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.113 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.044) 0:01:06.160 ***** 2020-11-19T03:25:47.120 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-19T03:25:47.121 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.157 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.204 ***** 2020-11-19T03:25:47.163 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T03:25:47.165 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.200 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.247 ***** 2020-11-19T03:25:47.206 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-19T03:25:47.208 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.244 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.291 ***** 2020-11-19T03:25:47.250 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-19T03:25:47.252 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.288 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.335 ***** 2020-11-19T03:25:47.294 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-19T03:25:47.296 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.333 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.378 ***** 2020-11-19T03:25:47.344 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.376 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.044) 0:01:06.423 ***** 2020-11-19T03:25:47.384 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-19T03:25:47.387 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.427 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.050) 0:01:06.474 ***** 2020-11-19T03:25:47.433 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-19T03:25:47.435 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.473 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.046) 0:01:06.520 ***** 2020-11-19T03:25:47.479 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:25:47.481 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.517 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.044) 0:01:06.564 ***** 2020-11-19T03:25:47.525 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-19T03:25:47.526 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.562 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.044) 0:01:06.609 ***** 2020-11-19T03:25:47.568 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-19T03:25:47.570 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.605 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.652 ***** 2020-11-19T03:25:47.612 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-19T03:25:47.614 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.648 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.042) 0:01:06.695 ***** 2020-11-19T03:25:47.662 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.698 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.050) 0:01:06.745 ***** 2020-11-19T03:25:47.705 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T03:25:47.706 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.742 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.789 ***** 2020-11-19T03:25:47.748 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-19T03:25:47.751 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.785 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.043) 0:01:06.832 ***** 2020-11-19T03:25:47.792 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.829 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.044) 0:01:06.876 ***** 2020-11-19T03:25:47.835 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-19T03:25:47.837 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.875 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.045) 0:01:06.922 ***** 2020-11-19T03:25:47.881 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-19T03:25:47.883 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.919 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.044) 0:01:06.966 ***** 2020-11-19T03:25:47.926 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-19T03:25:47.928 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:47.976 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:47 +0000 (0:00:00.056) 0:01:07.023 ***** 2020-11-19T03:25:47.983 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T03:25:47.985 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:48.019 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.043) 0:01:07.066 ***** 2020-11-19T03:25:48.026 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-19T03:25:48.028 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:48.063 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.043) 0:01:07.110 ***** 2020-11-19T03:25:48.069 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-19T03:25:48.079 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:48.107 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.044) 0:01:07.154 ***** 2020-11-19T03:25:48.114 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-19T03:25:48.117 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:48.154 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.046) 0:01:07.201 ***** 2020-11-19T03:25:48.181 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-19T03:25:48.184 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:48.219 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.065) 0:01:07.266 ***** 2020-11-19T03:25:48.246 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-19T03:25:48.248 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:48.286 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.066) 0:01:07.333 ***** 2020-11-19T03:25:48.293 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-19T03:25:48.293 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-19T03:25:48.336 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.050) 0:01:07.383 ***** 2020-11-19T03:25:48.343 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:48.380 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:48 +0000 (0:00:00.044) 0:01:07.427 ***** 2020-11-19T03:25:48.407 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-19T03:25:50.299 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.067) 0:01:07.495 ***** TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.063) 0:01:07.559 ***** TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.063) 0:01:07.622 ***** TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.071) 0:01:07.693 ***** TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.063) 0:01:07.757 ***** TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.044) 0:01:07.802 ***** TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.045) 2020-11-19T03:25:50.300 INFO:teuthology.task.ansible.out: 0:01:07.847 ***** TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.047) 0:01:07.895 ***** TASK [testnode : Set grub config.] ********************************************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:48 +0000 (0:00:00.045) 0:01:07.940 ***** TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} Thursday 19 November 2020 03:25:49 +0000 (0:00:00.172) 0:01:08.113 ***** TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} Thursday 19 November 2020 03:25:49 +0000 (0:00:00.232) 0:01:08.346 ***** TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} Thursday 19 November 2020 03:25:49 +0000 (0:00:00.154) 0:01:08.500 ***** TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:25:49 +0000 (0:00:00.063) 2020-11-19T03:25:50.302 INFO:teuthology.task.ansible.out: 0:01:08.563 ***** TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} Thursday 19 November 2020 03:25:49 +0000 (0:00:00.179) 0:01:08.743 ***** TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} Thursday 19 November 2020 03:25:49 +0000 (0:00:00.200) 0:01:08.943 ***** TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} Thursday 19 November 2020 03:25:50 +0000 (0:00:00.162) 0:01:09.105 ***** TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 19 November 2020 03:25:50 +0000 (0:00:00.170) 0:01:09.276 ***** TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file 2020-11-19T03:25:50.303 INFO:teuthology.task.ansible.out:", "uid": 0} Thursday 19 November 2020 03:25:50 +0000 (0:00:00.046) 0:01:09.323 ***** TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.321 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.045) 0:01:09.368 ***** 2020-11-19T03:25:50.348 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-19T03:25:50.350 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.385 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.062) 0:01:09.431 ***** 2020-11-19T03:25:50.411 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-19T03:25:50.413 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.447 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.062) 0:01:09.494 ***** 2020-11-19T03:25:50.474 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-19T03:25:50.475 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.514 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.067) 0:01:09.562 ***** 2020-11-19T03:25:50.541 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-19T03:25:50.543 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.580 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.062) 0:01:09.624 ***** 2020-11-19T03:25:50.583 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-19T03:25:50.585 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-19T03:25:50.622 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.044) 0:01:09.669 ***** 2020-11-19T03:25:50.629 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:25:50.630 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-19T03:25:50.666 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.044) 0:01:09.713 ***** 2020-11-19T03:25:50.704 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-19T03:25:50.706 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.716 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.727 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.738 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.749 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.751 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi066.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-19T03:25:50.789 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.122) 0:01:09.836 ***** 2020-11-19T03:25:50.796 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-19T03:25:50.798 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi154.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.799 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.799 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.799 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.801 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.812 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.813 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.850 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.060) 0:01:09.897 ***** 2020-11-19T03:25:50.857 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-19T03:25:50.858 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.901 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.050) 0:01:09.948 ***** 2020-11-19T03:25:50.907 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-19T03:25:50.909 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:50.950 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:50 +0000 (0:00:00.047) 0:01:09.995 ***** 2020-11-19T03:25:50.977 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-19T03:25:50.979 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.014 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.065) 0:01:10.061 ***** 2020-11-19T03:25:51.042 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-19T03:25:51.043 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.078 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.063) 0:01:10.125 ***** 2020-11-19T03:25:51.095 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi154.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.002981", "end": "2020-11-19 03:25:51.063508", "rc": 0, "start": "2020-11-19 03:25:51.060527", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-19T03:25:51.135 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.057) 0:01:10.182 ***** 2020-11-19T03:25:51.141 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-19T03:25:51.144 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.178 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.043) 0:01:10.225 ***** 2020-11-19T03:25:51.185 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-19T03:25:51.187 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.229 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.050) 0:01:10.276 ***** 2020-11-19T03:25:51.236 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:25:51.237 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-19T03:25:51.273 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.043) 0:01:10.320 ***** 2020-11-19T03:25:51.283 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.320 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.046) 0:01:10.367 ***** 2020-11-19T03:25:51.327 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-19T03:25:51.330 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.330 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.331 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.353 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-19T03:25:51.354 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi066.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.391 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.070) 0:01:10.438 ***** 2020-11-19T03:25:51.397 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-19T03:25:51.400 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.401 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.402 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.413 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.414 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.424 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.426 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.470 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.078) 0:01:10.517 ***** 2020-11-19T03:25:51.676 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-19T03:25:51.677 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003776", "end": "2020-11-19 03:25:51.652224", "rc": 0, "start": "2020-11-19 03:25:51.648448", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-19T03:25:51.723 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.253) 0:01:10.770 ***** 2020-11-19T03:25:51.762 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-19T03:25:51.764 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.803 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.079) 0:01:10.850 ***** 2020-11-19T03:25:51.830 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-19T03:25:51.832 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.867 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.063) 0:01:10.914 ***** 2020-11-19T03:25:51.899 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-19T03:25:51.901 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:51.936 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:51 +0000 (0:00:00.068) 0:01:10.983 ***** 2020-11-19T03:25:52.314 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:25:52.360 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.423) 0:01:11.406 ***** 2020-11-19T03:25:52.389 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:52.441 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.081) 0:01:11.488 ***** 2020-11-19T03:25:52.479 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-19T03:25:52.481 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item={'block_used': 13753459, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341649, 'size_available': 927095394304, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341649, "block_size": 4096, "block_total": 240095108, "block_used": 13753459, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095394304, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}} 2020-11-19T03:25:52.516 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.075) 0:01:11.563 ***** 2020-11-19T03:25:52.655 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-19T03:25:52.657 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009867", "end": "2020-11-19 03:25:52.635459", "rc": 0, "start": "2020-11-19 03:25:52.625592", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:25:52.702 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.185) 0:01:11.749 ***** 2020-11-19T03:25:52.740 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-19T03:25:52.742 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'block_used': 13753459, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341649, 'size_available': 927095394304, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341649, "block_size": 4096, "block_total": 240095108, "block_used": 13753459, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095394304, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:52.794 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.092) 0:01:11.841 ***** 2020-11-19T03:25:52.821 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-19T03:25:52.857 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.063) 0:01:11.905 ***** 2020-11-19T03:25:52.864 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-19T03:25:52.866 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:25:52.906 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.045) 0:01:11.950 ***** 2020-11-19T03:25:52.912 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-19T03:25:52.914 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:52.924 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-19T03:25:52.926 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:52.960 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:52 +0000 (0:00:00.056) 0:01:12.006 ***** 2020-11-19T03:25:52.997 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-19T03:25:52.999 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item={'block_used': 13753456, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341652, 'size_available': 927095406592, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341652, "block_size": 4096, "block_total": 240095108, "block_used": 13753456, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095406592, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}} 2020-11-19T03:25:53.034 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:53 +0000 (0:00:00.074) 0:01:12.081 ***** 2020-11-19T03:25:53.170 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009657", "end": "2020-11-19 03:25:53.148760", "rc": 0, "start": "2020-11-19 03:25:53.139103", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:25:53.222 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:53 +0000 (0:00:00.187) 0:01:12.269 ***** 2020-11-19T03:25:53.265 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-19T03:25:53.267 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'block_used': 13753456, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341652, 'size_available': 927095406592, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341652, "block_size": 4096, "block_total": 240095108, "block_used": 13753456, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095406592, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:53.303 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:25:53 +0000 (0:00:00.081) 0:01:12.350 ***** 2020-11-19T03:25:53.822 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** Thursday 19 November 2020 03:25:53 +0000 (0:00:00.061) 0:01:12.412 ***** TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:55.067 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPH5SXuluACT8fI5g2s2EF2gpRN0HvFe3F'], 'uuids': ['8166798a-2867-4a29-9669-8c0d57764516']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.046216", "end": "2020-11-19 03:25:55.039710", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPH5SXuluACT8fI5g2s2EF2gpRN0HvFe3F"], "labels": [], "masters": [], "uuids": ["8166798a-2867-4a29-9669-8c0d57764516"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:52.993494", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:55.078 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:55.089 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:55.510 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XwAlDG2JUI90L3cuc6gcxfsxjOuGLhfBM'], 'uuids': ['080deb0f-d10e-46b4-a10b-5798da2167fa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.952432", "end": "2020-11-19 03:25:55.485482", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XwAlDG2JUI90L3cuc6gcxfsxjOuGLhfBM"], "labels": [], "masters": [], "uuids": ["080deb0f-d10e-46b4-a10b-5798da2167fa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:53.533050", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:55.522 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:55.533 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:56.335 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPMVI8KfqBwDiCklxWaBkjCUCC7Zs5TKP4'], 'uuids': ['13cefefd-b9d5-4cff-99aa-d1dacb159282']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.134205", "end": "2020-11-19 03:25:56.316149", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPMVI8KfqBwDiCklxWaBkjCUCC7Zs5TKP4"], "labels": [], "masters": [], "uuids": ["13cefefd-b9d5-4cff-99aa-d1dacb159282"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:55.181944", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:56.349 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL', 'wwn-0x5000c50092924855'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1', 'wwn-0x5000c50092924855-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092924855', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFNL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL", "wwn-0x5000c50092924855"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1", "wwn-0x5000c50092924855-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924855"}}, "skip_reason": 2020-11-19T03:25:56.354 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T03:25:56.760 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xv8Nep2xREPNOPOX93V1mAr6fdIBBeEH5'], 'uuids': ['0c3b574a-225b-487b-be53-48b9d9950a71']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.111126", "end": "2020-11-19 03:25:56.742089", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xv8Nep2xREPNOPOX93V1mAr6fdIBBeEH5"], "labels": [], "masters": [], "uuids": ["0c3b574a-225b-487b-be53-48b9d9950a71"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:55.630963", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:56.774 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2RJ', 'wwn-0x5000c50087b90a26'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2RJ-part1', 'wwn-0x5000c50087b90a26-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b90a26', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2RJ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2RJ", "wwn-0x5000c50087b90a26"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2RJ-part1", "wwn-0x5000c50087b90a26-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2RJ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b90a26"}}, "skip_reason": 2020-11-19T03:25:56.776 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T03:25:57.707 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ptG2VI-O80H-41sR-7BNE-q5xR-qc57-B04eTS', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN', 'nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.165168", "end": "2020-11-19 03:25:57.673097", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ptG2VI-O80H-41sR-7BNE-q5xR-qc57-B04eTS", "nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN", "nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:56.507929", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-19T03:25:57.709 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:57.709 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:57.722 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:58.043 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-33zwBr-cV7D-oAMH-wHFN-0YYq-wJeK-tHSTiz', 'nvme-INTEL_SSDPEDMD400G4_CVFT53430091400BGN', 'nvme-nvme.8086-43564654353334333030393134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.152427", "end": "2020-11-19 03:25:58.022879", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-33zwBr-cV7D-oAMH-wHFN-0YYq-wJeK-tHSTiz", "nvme-INTEL_SSDPEDMD400G4_CVFT53430091400BGN", "nvme-nvme.8086-43564654353334333030393134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:56.870452", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-19T03:25:58.044 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:58.054 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:58.065 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:59.020 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPc0fVk3NqvBfjjKVtAYLqF9ZGEZTScJLK'], 'uuids': ['461b427e-4076-43fc-bee0-330c6584cc2d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.164002", "end": "2020-11-19 03:25:58.990706", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPc0fVk3NqvBfjjKVtAYLqF9ZGEZTScJLK"], "labels": [], "masters": [], "uuids": ["461b427e-4076-43fc-bee0-330c6584cc2d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:57.826704", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:59.031 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:59.043 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:59.350 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XWCcXDfXRtDCxbhUPeCCeeUI3zQRG0Xw9'], 'uuids': ['8bb7d29a-62a0-4807-943f-f591b5a2b92d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.136870", "end": "2020-11-19 03:25:59.300362", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XWCcXDfXRtDCxbhUPeCCeeUI3zQRG0Xw9"], "labels": [], "masters": [], "uuids": ["8bb7d29a-62a0-4807-943f-f591b5a2b92d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:58.163492", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:25:59.361 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:25:59.372 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:00.392 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPVpCFwRiIf0bkQYLZ7Hj6DSsghyYPxXCL'], 'uuids': ['1ae2fe5b-01f3-45e4-a95c-109c70d4970e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.226193", "end": "2020-11-19 03:26:00.366784", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPVpCFwRiIf0bkQYLZ7Hj6DSsghyYPxXCL"], "labels": [], "masters": [], "uuids": ["1ae2fe5b-01f3-45e4-a95c-109c70d4970e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:59.140591", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:26:00.405 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:00.628 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XCefynakowm3mDhsAyLM1n67GrBm548kh'], 'uuids': ['e050948c-518c-4a06-a008-fc17093633c7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.124213", "end": "2020-11-19 03:26:00.600445", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XCefynakowm3mDhsAyLM1n67GrBm548kh"], "labels": [], "masters": [], "uuids": ["e050948c-518c-4a06-a008-fc17093633c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:25:59.476232", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:26:00.640 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:01.679 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPhqTCdEH4S4tfm45QVsE90Ox0DMRbcrHU'], 'uuids': ['2aef6c5c-1807-4d50-902e-3c00a178d165']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.152830", "end": "2020-11-19 03:26:01.650613", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPhqTCdEH4S4tfm45QVsE90Ox0DMRbcrHU"], "labels": [], "masters": [], "uuids": ["2aef6c5c-1807-4d50-902e-3c00a178d165"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:00.497783", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:26:01.739 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:01 +0000 (0:00:08.374) 0:01:20.786 ***** 2020-11-19T03:26:01.801 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-19T03:26:01.803 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:01.879 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-19T03:26:01.881 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xoc0mdFeuYWUK4FHqxykABn7fR0BfIkIJ'], 'uuids': ['0adb946d-8183-4636-8294-3fd71311eefe']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.108566", "end": "2020-11-19 03:26:01.858795", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xoc0mdFeuYWUK4FHqxykABn7fR0BfIkIJ"], "labels": [], "masters": [], "uuids": ["0adb946d-8183-4636-8294-3fd71311eefe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:00.750229", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-19T03:26:01.931 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:01 +0000 (0:00:00.191) 0:01:20.978 ***** 2020-11-19T03:26:01.972 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-19T03:26:01.974 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.091 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPH5SXuluACT8fI5g2s2EF2gpRN0HvFe3F'], 'uuids': ['8166798a-2867-4a29-9669-8c0d57764516']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.166266", "end": "2020-11-19 03:26:02.070838", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPH5SXuluACT8fI5g2s2EF2gpRN0HvFe3F"], "labels": [], "masters": [], "uuids": ["8166798a-2867-4a29-9669-8c0d57764516"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:01.904572", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161536 s, 714 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161536 s, 714 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:02.103 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.116 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.267 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XwAlDG2JUI90L3cuc6gcxfsxjOuGLhfBM'], 'uuids': ['080deb0f-d10e-46b4-a10b-5798da2167fa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153190", "end": "2020-11-19 03:26:02.235623", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XwAlDG2JUI90L3cuc6gcxfsxjOuGLhfBM"], "labels": [], "masters": [], "uuids": ["080deb0f-d10e-46b4-a10b-5798da2167fa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:02.082433", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148379 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148379 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:02.269 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.282 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.401 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPMVI8KfqBwDiCklxWaBkjCUCC7Zs5TKP4'], 'uuids': ['13cefefd-b9d5-4cff-99aa-d1dacb159282']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.160594", "end": "2020-11-19 03:26:02.367937", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPMVI8KfqBwDiCklxWaBkjCUCC7Zs5TKP4"], "labels": [], "masters": [], "uuids": ["13cefefd-b9d5-4cff-99aa-d1dacb159282"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:02.207343", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155731 s, 741 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155731 s, 741 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:02.416 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL', 'wwn-0x5000c50092924855'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1', 'wwn-0x5000c50092924855-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092924855', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFNL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL", "wwn-0x5000c50092924855"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFNL-part1", "wwn-0x5000c50092924855-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924855"}}, "skip_reason": 2020-11-19T03:26:02.418 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T03:26:02.581 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xv8Nep2xREPNOPOX93V1mAr6fdIBBeEH5'], 'uuids': ['0c3b574a-225b-487b-be53-48b9d9950a71']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.156668", "end": "2020-11-19 03:26:02.545910", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xv8Nep2xREPNOPOX93V1mAr6fdIBBeEH5"], "labels": [], "masters": [], "uuids": ["0c3b574a-225b-487b-be53-48b9d9950a71"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:02.389242", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151955 s, 759 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151955 s, 759 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:02.584 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2RJ', 'wwn-0x5000c50087b90a26'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2RJ-part1', 'wwn-0x5000c50087b90a26-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b90a26', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2RJ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2RJ", "wwn-0x5000c50087b90a26"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2RJ-part1", "wwn-0x5000c50087b90a26-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2RJ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b90a26"}}, "skip_reason": 2020-11-19T03:26:02.586 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-19T03:26:02.618 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ptG2VI-O80H-41sR-7BNE-q5xR-qc57-B04eTS', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN', 'nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053407", "end": "2020-11-19 03:26:02.585891", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ptG2VI-O80H-41sR-7BNE-q5xR-qc57-B04eTS", "nvme-INTEL_SSDPEDMD400G4_CVFT6233003B400BGN", "nvme-nvme.8086-43564654363233333030334234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:02.532484", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0486604 s, 2.4 GB/s", "std 2020-11-19T03:26:02.620 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0486604 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:02.630 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.641 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.773 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-33zwBr-cV7D-oAMH-wHFN-0YYq-wJeK-tHSTiz', 'nvme-INTEL_SSDPEDMD400G4_CVFT53430091400BGN', 'nvme-nvme.8086-43564654353334333030393134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.055478", "end": "2020-11-19 03:26:02.738829", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-33zwBr-cV7D-oAMH-wHFN-0YYq-wJeK-tHSTiz", "nvme-INTEL_SSDPEDMD400G4_CVFT53430091400BGN", "nvme-nvme.8086-43564654353334333030393134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:02.683351", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0508464 s, 2.3 GB/s", "std 2020-11-19T03:26:02.775 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0508464 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.783 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.922 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPc0fVk3NqvBfjjKVtAYLqF9ZGEZTScJLK'], 'uuids': ['461b427e-4076-43fc-bee0-330c6584cc2d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.150432", "end": "2020-11-19 03:26:02.894212", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPc0fVk3NqvBfjjKVtAYLqF9ZGEZTScJLK"], "labels": [], "masters": [], "uuids": ["461b427e-4076-43fc-bee0-330c6584cc2d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:02.743780", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147701 s, 781 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147701 s, 781 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:02.933 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:02.945 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:03.062 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XWCcXDfXRtDCxbhUPeCCeeUI3zQRG0Xw9'], 'uuids': ['8bb7d29a-62a0-4807-943f-f591b5a2b92d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149601", "end": "2020-11-19 03:26:03.031641", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XWCcXDfXRtDCxbhUPeCCeeUI3zQRG0Xw9"], "labels": [], "masters": [], "uuids": ["8bb7d29a-62a0-4807-943f-f591b5a2b92d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:02.882040", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146415 s, 788 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146415 s, 788 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:03.073 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:03.096 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:03.225 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPVpCFwRiIf0bkQYLZ7Hj6DSsghyYPxXCL'], 'uuids': ['1ae2fe5b-01f3-45e4-a95c-109c70d4970e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.160512", "end": "2020-11-19 03:26:03.201457", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPVpCFwRiIf0bkQYLZ7Hj6DSsghyYPxXCL"], "labels": [], "masters": [], "uuids": ["1ae2fe5b-01f3-45e4-a95c-109c70d4970e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:03.040945", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156173 s, 739 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156173 s, 739 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:03.237 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:03.387 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XCefynakowm3mDhsAyLM1n67GrBm548kh'], 'uuids': ['e050948c-518c-4a06-a008-fc17093633c7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156952", "end": "2020-11-19 03:26:03.355108", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21XCefynakowm3mDhsAyLM1n67GrBm548kh"], "labels": [], "masters": [], "uuids": ["e050948c-518c-4a06-a008-fc17093633c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:03.198156", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.1521 s, 758 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.1521 s, 758 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:03.400 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:03.539 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPhqTCdEH4S4tfm45QVsE90Ox0DMRbcrHU'], 'uuids': ['2aef6c5c-1807-4d50-902e-3c00a178d165']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.162672", "end": "2020-11-19 03:26:03.511128", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-aw756xHrDNxoXNYmEMl8zsRosGLADCHPhqTCdEH4S4tfm45QVsE90Ox0DMRbcrHU"], "labels": [], "masters": [], "uuids": ["2aef6c5c-1807-4d50-902e-3c00a178d165"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:03.348456", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.157956 s, 730 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.157956 s, 730 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:05.407 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:03 +0000 (0:00:01.655) 0:01:22.634 ***** changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xoc0mdFeuYWUK4FHqxykABn7fR0BfIkIJ'], 'uuids': ['0adb946d-8183-4636-8294-3fd71311eefe']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.157040", "end": "2020-11-19 03:26:03.663883", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6HR2hLVFmQadcZBroT7cpfwc1iLpa21Xoc0mdFeuYWUK4FHqxykABn7fR0BfIkIJ"], "labels": [], "masters": [], "uuids": ["0adb946d-8183-4636-8294-3fd71311eefe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-19 03:26:03.506843", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152191 s, 758 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152191 s, 758 MB/s"], "stdout": "", "stdout_lines": []} Thursday 19 November 2020 03:26:03 +0000 (0:00:00.156) 0:01:22.790 ***** TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi154.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.159582", "end": "2020-11-19 03: 2020-11-19T03:26:05.409 INFO:teuthology.task.ansible.out:26:03.873043", "rc": 0, "start": "2020-11-19 03:26:03.713461", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 19 November 2020 03:26:03 +0000 (0:00:00.210) 0:01:23.001 ***** changed: [smithi066.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.173512", "end": "2020-11-19 03:26:04.022805", "rc": 0, "start": "2020-11-19 03:26:03.849293", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.151) 0:01:23.153 ***** TASK [testnode : Check for physical volumes] *********************************** changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.059556", "end": "2020-11-19 03:26:04.131422", "rc": 0, "start": "2020-11-19 03:26:04.071866", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.101) 0:01:23.255 ***** TASK [testnode : Remove physical volumes] ************************************** Thursday 19 November 2020 03:26:04 +0000 (0:00:00.067) 0:01:23.323 ***** TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.065) 0:01:23.388 ***** TASK [testnode : Check for physical volumes] *********************************** changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.070252", "end": "2020-11-19 03:26:04.293340", "rc": 0, "start": "2020-11-19 03:26:04.223088", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.044) 0:01:23.432 ***** TASK [testnode : Write pa 2020-11-19T03:26:05.411 INFO:teuthology.task.ansible.out:rtition entries to {{ item.0.device }}] *************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.044) 0:01:23.477 ***** TASK [testnode : Remove physical volumes] ************************************** Thursday 19 November 2020 03:26:04 +0000 (0:00:00.046) 0:01:23.523 ***** TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.046) 0:01:23.570 ***** TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.046) 0:01:23.616 ***** TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.044) 0:01:23.660 ***** TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.047) 0:01:23.707 ***** TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.084) 0:01:23.792 ***** TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 19 Novemb 2020-11-19T03:26:05.413 INFO:teuthology.task.ansible.out:er 2020 03:26:04 +0000 (0:00:00.045) 0:01:23.838 ***** TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.044) 0:01:23.883 ***** TASK [testnode : Set root disk] ************************************************ skipping: [smithi154.front.sepia.ceph.com] => (item={'block_used': 13753459, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341649, 'size_available': 927095394304, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341649, "block_size": 4096, "block_total": 240095108, "block_used": 13753459, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095394304, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.047) 0:01:23.930 ***** TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:04 +0000 (0:00:00.064) 0:01:23.995 ***** TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi066.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : set_fact] ***************************************************** skip 2020-11-19T03:26:05.415 INFO:teuthology.task.ansible.out:ping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.063) 0:01:24.058 ***** Thursday 19 November 2020 03:26:05 +0000 (0:00:00.034) 0:01:24.093 ***** TASK [testnode : Set root disk] ************************************************ skipping: [smithi066.front.sepia.ceph.com] => (item={'block_used': 13753456, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341652, 'size_available': 927095406592, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341652, "block_size": 4096, "block_total": 240095108, "block_used": 13753456, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095406592, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.045) 0:01:24.139 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.044) 0:01:24.184 ***** TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.044) 0:01:24.228 ***** TASK [testnode : Create volume_groups dict] ** 2020-11-19T03:26:05.417 INFO:teuthology.task.ansible.out:********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.044) 0:01:24.273 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.044) 0:01:24.317 ***** TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.045) 0:01:24.362 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:05 +0000 (0:00:00.044) 0:01:24.407 ***** TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:05.418 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:05 +0000 (0:00:00.057) 0:01:24.465 ***** 2020-11-19T03:26:05.426 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-19T03:26:05.428 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:05.488 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:05 +0000 (0:00:00.070) 0:01:24.535 ***** 2020-11-19T03:26:05.516 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:05.560 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:05 +0000 (0:00:00.071) 0:01:24.607 ***** 2020-11-19T03:26:05.589 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-19T03:26:05.590 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:05.628 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:05 +0000 (0:00:00.068) 0:01:24.675 ***** 2020-11-19T03:26:06.014 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-19T03:26:06.016 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-19T03:26:06.051 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:06 +0000 (0:00:00.423) 0:01:25.098 ***** 2020-11-19T03:26:06.058 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-19T03:26:06.096 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:06 +0000 (0:00:00.045) 0:01:25.143 ***** 2020-11-19T03:26:06.602 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-19T03:26:06.606 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:06.863 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:06.899 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:07.090 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:07.144 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:07.335 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:07.378 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:07.576 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:07.625 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:07 +0000 (0:00:01.528) 0:01:26.672 ***** 2020-11-19T03:26:07.632 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-19T03:26:07.674 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:07 +0000 (0:00:00.048) 0:01:26.720 ***** 2020-11-19T03:26:07.774 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-19T03:26:07.777 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-19T03:26:07.828 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:07 +0000 (0:00:00.154) 0:01:26.874 ***** 2020-11-19T03:26:07.859 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-19T03:26:07.905 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:07 +0000 (0:00:00.077) 0:01:26.952 ***** 2020-11-19T03:26:07.980 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-19T03:26:07.981 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T03:26:08.056 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T03:26:08.100 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T03:26:08.178 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T03:26:08.220 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T03:26:08.324 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T03:26:08.351 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-19T03:26:08.374 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:08.417 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:08 +0000 (0:00:00.510) 0:01:27.463 ***** 2020-11-19T03:26:10.059 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} skipping: [smithi066.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:08 +0000 (0:00:00.104) 0:01:27.568 ***** TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} Thursday 19 November 2020 03:26:08 +0000 (0:00:00.090) 0:01:27.658 ***** TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} Thursday 19 November 2020 03:26:08 +0000 (0:00:00.076) 0:01:27.735 ***** TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} Thursday 19 November 2020 03:26:08 +0000 (0:00:00.045) 0:01:27.780 ***** TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} Thursday 19 November 2020 03:26:08 +0000 (0:00:00.091) 0:01:27.871 ***** TASK [testnode : Create xfs filesystem on /dev/vg_nvm 2020-11-19T03:26:10.061 INFO:teuthology.task.ansible.out:e/lv_5] ******************* changed: [smithi066.front.sepia.ceph.com] => {"changed": true} Thursday 19 November 2020 03:26:09 +0000 (0:00:00.383) 0:01:28.255 ***** TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} Thursday 19 November 2020 03:26:09 +0000 (0:00:00.494) 0:01:28.749 ***** TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} Thursday 19 November 2020 03:26:09 +0000 (0:00:00.192) 0:01:28.941 ***** TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T03:26:10.071 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:10 +0000 (0:00:00.176) 0:01:29.118 ***** 2020-11-19T03:26:10.186 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-19T03:26:10.188 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true} 2020-11-19T03:26:10.225 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:10 +0000 (0:00:00.153) 0:01:29.272 ***** 2020-11-19T03:26:10.401 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-19T03:26:10.402 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-19T03:26:10.442 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:10 +0000 (0:00:00.217) 0:01:29.489 ***** 2020-11-19T03:26:10.449 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-19T03:26:10.451 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-19T03:26:10.487 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:10 +0000 (0:00:00.044) 0:01:29.534 ***** 2020-11-19T03:26:10.619 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-19T03:26:10.621 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-19T03:26:10.657 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:10 +0000 (0:00:00.170) 0:01:29.704 ***** 2020-11-19T03:26:10.760 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-19T03:26:10.762 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-19T03:26:10.799 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:10 +0000 (0:00:00.141) 0:01:29.846 ***** 2020-11-19T03:26:10.806 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-19T03:26:10.807 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-19T03:26:10.844 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:10 +0000 (0:00:00.044) 0:01:29.891 ***** 2020-11-19T03:26:11.174 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-19T03:26:11.212 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:11 +0000 (0:00:00.367) 0:01:30.259 ***** 2020-11-19T03:26:11.548 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-19T03:26:11.550 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-19T03:26:11.585 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:11 +0000 (0:00:00.372) 0:01:30.631 ***** 2020-11-19T03:26:11.720 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-19T03:26:11.722 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:26:11.758 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:11 +0000 (0:00:00.174) 0:01:30.805 ***** 2020-11-19T03:26:11.947 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-19T03:26:11.949 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.049339", "end": "2020-11-19 03:26:11.926666", "rc": 0, "start": "2020-11-19 03:26:11.877327", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-19T03:26:11.984 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:11 +0000 (0:00:00.226) 0:01:31.031 ***** 2020-11-19T03:26:12.011 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-19T03:26:12.013 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:12.050 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:12 +0000 (0:00:00.064) 0:01:31.096 ***** 2020-11-19T03:26:12.082 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:12.118 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:12 +0000 (0:00:00.068) 0:01:31.165 ***** 2020-11-19T03:26:12.526 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-19T03:26:12.528 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:26:12.563 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:12 +0000 (0:00:00.445) 0:01:31.610 ***** 2020-11-19T03:26:12.758 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-19T03:26:12.760 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045370", "end": "2020-11-19 03:26:12.730344", "rc": 0, "start": "2020-11-19 03:26:12.684974", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-19T03:26:12.795 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:12 +0000 (0:00:00.231) 0:01:31.842 ***** 2020-11-19T03:26:12.833 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-19T03:26:12.835 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:12.877 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:12 +0000 (0:00:00.082) 0:01:31.924 ***** 2020-11-19T03:26:12.904 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:12.940 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:12 +0000 (0:00:00.063) 0:01:31.987 ***** 2020-11-19T03:26:13.009 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-19T03:26:13.011 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:26:13.047 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:13 +0000 (0:00:00.106) 0:01:32.094 ***** 2020-11-19T03:26:13.075 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-19T03:26:13.076 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:13.116 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:13 +0000 (0:00:00.068) 0:01:32.163 ***** 2020-11-19T03:26:13.492 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-19T03:26:15.043 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} Thursday 19 November 2020 03:26:13 +0000 (0:00:00.412) 0:01:32.576 ***** TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:13 +0000 (0:00:00.071) 0:01:32.647 ***** TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} Thursday 19 November 2020 03:26:13 +0000 (0:00:00.223) 0:01:32.871 ***** TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 03:24:22 UTC", "ActiveEnterTimestampMonotonic": "27418074", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount sysinit.target systemd-journald.socket basic.target network.target system.slice systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 03:24:15 UTC", "AssertTimestampMonotonic": "20980236", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanSt 2020-11-19T03:26:15.045 INFO:teuthology.task.ansible.out:op": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 03:24:15 UTC", "ConditionTimestampMonotonic": "20980235", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1632", "ExecMainStartTimestamp": "Thu 2020-11-19 03:24:22 UTC", "ExecMainStartTimestampMonotonic": "27418060", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "Inac 2020-11-19T03:26:15.047 INFO:teuthology.task.ansible.out:tiveExitTimestamp": "Thu 2020-11-19 03:24:15 UTC", "InactiveExitTimestampMonotonic": "20981622", "InvocationID": "e75c20a2a34f49d1b96f3387a49de5af", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1632", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKer 2020-11-19T03:26:15.049 INFO:teuthology.task.ansible.out:nelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 03:24:22 UTC", "StateChangeTimestampMonotonic": "27418074", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 03:24:22 UTC", "WatchdogTimestampMonotonic": "27418073", "WatchdogUSec": "0"}} Thursday 19 November 2020 03:26:13 +0000 (0:00:00.049) 0:01:32.920 ***** TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi154.fro 2020-11-19T03:26:15.050 INFO:teuthology.task.ansible.out:nt.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:13 +0000 (0:00:00.043) 0:01:32.964 ***** TASK [testnode : set_fact] ***************************************************** ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} Thursday 19 November 2020 03:26:13 +0000 (0:00:00.044) 0:01:33.009 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:14 +0000 (0:00:00.064) 0:01:33.073 ***** TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:14 +0000 (0:00:00.083) 0:01:33.156 ***** TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} Thursday 19 November 2020 03:26:14 +0000 (0:00:00.244) 0:01:33.401 ***** TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:14 +0000 (0:00:00.099) 0:01:33.501 ***** TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-11-19 03:24:34 UTC", 2020-11-19T03:26:15.053 INFO:teuthology.task.ansible.out: "ActiveEnterTimestampMonotonic": "27586268", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service -.mount network.target systemd-journald.socket basic.target sysinit.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-11-19 03:24:26 UTC", "AssertTimestampMonotonic": "21008061", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-11-19 03:24:26 UTC", "ConditionTimestampMonotonic": "21008060", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1853", "ExecMainStartTimestamp": "Thu 2020-11-19 03:24: 2020-11-19T03:26:15.055 INFO:teuthology.task.ansible.out:34 UTC", "ExecMainStartTimestampMonotonic": "27586256", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-11-19 03:24:26 UTC", "InactiveExitTimestampMonotonic": "21008793", "InvocationID": "a5e4aae2672f4c2191da08202d8d6b90", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadS 2020-11-19T03:26:15.057 INFO:teuthology.task.ansible.out:tate": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1853", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inh 2020-11-19T03:26:15.057 INFO:teuthology.task.ansible.out:erit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-11-19 03:24:34 UTC", "StateChangeTimestampMonotonic": "27586268", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFa 2020-11-19T03:26:15.060 INFO:teuthology.task.ansible.out:cility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-11-19 03:24:34 UTC", "WatchdogTimestampMonotonic": "27586267", "WatchdogUSec": "0"}} TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi066.front.sepia.ceph.com] => {"cache_update_time": 1605756058, "cache_updated": false, "changed": false} 2020-11-19T03:26:15.096 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.641) 0:01:34.143 ***** 2020-11-19T03:26:15.131 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.035) 0:01:34.178 ***** 2020-11-19T03:26:15.137 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-19T03:26:15.139 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:15.174 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.043) 0:01:34.222 ***** 2020-11-19T03:26:15.181 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:26:15.183 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-19T03:26:15.220 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.045) 0:01:34.267 ***** 2020-11-19T03:26:15.247 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:26:15.248 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:15.284 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.063) 0:01:34.331 ***** 2020-11-19T03:26:15.311 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-19T03:26:15.313 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:15.349 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.065) 0:01:34.397 ***** 2020-11-19T03:26:15.385 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.035) 0:01:34.432 ***** 2020-11-19T03:26:15.647 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-19T03:26:15.649 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-19T03:26:15.693 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:15 +0000 (0:00:00.308) 0:01:34.740 ***** 2020-11-19T03:26:16.250 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi154.front.sepia.ceph.com] => {"cache_update_time": 1605756102, "cache_updated": false, "changed": false} 2020-11-19T03:26:16.289 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:16 +0000 (0:00:00.595) 0:01:35.336 ***** 2020-11-19T03:26:16.316 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:16.352 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:16 +0000 (0:00:00.063) 0:01:35.399 ***** 2020-11-19T03:26:16.483 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-19T03:26:16.488 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-19T03:26:16.520 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:16 +0000 (0:00:00.168) 0:01:35.567 ***** 2020-11-19T03:26:16.727 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-19T03:26:16.728 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-19T03:26:16.765 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:16 +0000 (0:00:00.244) 0:01:35.812 ***** 2020-11-19T03:26:16.835 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-19T03:26:16.836 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-19T03:26:16.872 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:16 +0000 (0:00:00.107) 0:01:35.919 ***** 2020-11-19T03:26:17.312 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi066.front.sepia.ceph.com] => {"changed": false} 2020-11-19T03:26:17.349 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:17 +0000 (0:00:00.477) 0:01:36.396 ***** 2020-11-19T03:26:17.377 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-19T03:26:17.415 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:17 +0000 (0:00:00.065) 0:01:36.462 ***** 2020-11-19T03:26:17.603 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-19T03:26:17.605 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-19T03:26:17.640 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:17 +0000 (0:00:00.225) 0:01:36.687 ***** 2020-11-19T03:26:17.650 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1605756029.0484774, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1605756029.0444775, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2817, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1605756029.0444775, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2282523401", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-19T03:26:17.689 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:17 +0000 (0:00:00.048) 0:01:36.736 ***** 2020-11-19T03:26:18.062 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-19T03:26:18.063 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-19T03:26:18.099 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:18 +0000 (0:00:00.410) 0:01:37.146 ***** 2020-11-19T03:26:18.126 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-19T03:26:18.128 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-19T03:26:18.166 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:18 +0000 (0:00:00.064) 0:01:37.211 ***** 2020-11-19T03:26:18.182 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi154.front.sepia.ceph.com] => {"changed": false} 2020-11-19T03:26:18.220 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:18 +0000 (0:00:00.056) 0:01:37.268 ***** 2020-11-19T03:26:18.223 INFO:teuthology.task.ansible.out: 2020-11-19T03:26:18.229 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-19T03:26:18.231 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:18.274 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:18 +0000 (0:00:00.053) 0:01:37.321 ***** 2020-11-19T03:26:18.285 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-19T03:26:18.286 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-19T03:26:18.333 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:18 +0000 (0:00:00.059) 0:01:37.380 ***** 2020-11-19T03:26:18.415 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-19T03:26:18.958 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} Thursday 19 November 2020 03:26:18 +0000 (0:00:00.122) 0:01:37.502 ***** TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1605756038.328037, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1605756038.324037, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2817, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1605756038.324037, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2282523401", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} Thursday 19 November 2020 03:26:18 +0000 (0:00:00.091) 0:01:37.594 ***** TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi066.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} Thursday 19 November 2020 03:26:18 +0000 (0:00:00.096) 0:01:37.691 ***** TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:18 +0000 (0:00:00.064) 0:01:37.756 ***** TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi154.front.sepia.ceph.com] => {"changed": false, "msg": ""} Thursday 19 November 2020 03:26:18 +0000 (0:00:00.046) 0:01:37.802 ***** TASK [testnode : grep ethtool f 2020-11-19T03:26:18.960 INFO:teuthology.task.ansible.out:or secondary NIC MAC address] ******************* skipping: [smithi066.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi154.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi066.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:18 +0000 (0:00:00.078) 0:01:37.881 ***** Thursday 19 November 2020 03:26:18 +0000 (0:00:00.034) 0:01:37.916 ***** TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi066.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was 2020-11-19T03:26:18.961 INFO:teuthology.task.ansible.out:False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi066.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi066.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} Thursday 19 November 2020 03:26:18 +0000 (0:00:00.089) 0:01:38.005 ***** 2020-11-19T03:26:19.004 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.045) 0:01:38.051 ***** 2020-11-19T03:26:19.010 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-19T03:26:19.012 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.054 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.049) 0:01:38.101 ***** 2020-11-19T03:26:19.083 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.126 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.069) 0:01:38.171 ***** 2020-11-19T03:26:19.142 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-19T03:26:19.146 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-19T03:26:19.183 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.059) 0:01:38.230 ***** 2020-11-19T03:26:19.190 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-19T03:26:19.193 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.230 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.046) 0:01:38.277 ***** 2020-11-19T03:26:19.259 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.294 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.064) 0:01:38.341 ***** 2020-11-19T03:26:19.322 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-19T03:26:19.323 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.360 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.066) 0:01:38.407 ***** 2020-11-19T03:26:19.402 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.040) 0:01:38.448 ***** 2020-11-19T03:26:19.412 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-19T03:26:19.414 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.460 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.059) 0:01:38.507 ***** 2020-11-19T03:26:19.468 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.505 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.044) 0:01:38.552 ***** 2020-11-19T03:26:19.512 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-19T03:26:19.514 INFO:teuthology.task.ansible.out:skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.557 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.052) 0:01:38.604 ***** 2020-11-19T03:26:19.565 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-19T03:26:19.568 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.577 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.579 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi066.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.614 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.056) 0:01:38.661 ***** 2020-11-19T03:26:19.621 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-19T03:26:19.622 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.659 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.045) 0:01:38.706 ***** 2020-11-19T03:26:19.698 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-19T03:26:19.700 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.709 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.711 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.721 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi154.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.759 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.099) 0:01:38.806 ***** 2020-11-19T03:26:19.786 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-19T03:26:19.788 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010838", "end": "2020-11-19 03:26:19.745939", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-19 03:26:19.735101", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : set_fact] ***************************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.825 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.065) 0:01:38.872 ***** 2020-11-19T03:26:19.852 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-19T03:26:19.854 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.889 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.064) 0:01:38.936 ***** 2020-11-19T03:26:19.906 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-19T03:26:19.908 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006012", "end": "2020-11-19 03:26:19.871020", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-19 03:26:19.865008", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:19.918 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:19.957 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:19 +0000 (0:00:00.067) 0:01:39.004 ***** 2020-11-19T03:26:19.984 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-19T03:26:19.986 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:20.021 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:20 +0000 (0:00:00.064) 0:01:39.068 ***** 2020-11-19T03:26:20.027 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-19T03:26:20.029 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005529", "end": "2020-11-19 03:26:19.984310", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-19 03:26:19.978781", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:20.050 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-19T03:26:20.052 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:20.088 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:20 +0000 (0:00:00.067) 0:01:39.135 ***** 2020-11-19T03:26:20.115 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-19T03:26:20.117 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:20.157 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:20 +0000 (0:00:00.068) 0:01:39.204 ***** 2020-11-19T03:26:20.164 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-19T03:26:20.166 INFO:teuthology.task.ansible.out:ok: [smithi066.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.021490", "end": "2020-11-19 03:26:20.116958", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-19 03:26:20.095468", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:20.205 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:20 +0000 (0:00:00.047) 0:01:39.252 ***** 2020-11-19T03:26:20.211 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-19T03:26:20.213 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:20.253 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:20 +0000 (0:00:00.048) 0:01:39.300 ***** 2020-11-19T03:26:20.260 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-19T03:26:20.296 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:20 +0000 (0:00:00.042) 0:01:39.343 ***** 2020-11-19T03:26:20.303 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-19T03:26:20.305 INFO:teuthology.task.ansible.out:skipping: [smithi154.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-19T03:26:20.458 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:20 +0000 (0:00:00.161) 0:01:39.505 ***** 2020-11-19T03:26:20.465 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-19T03:26:20.466 INFO:teuthology.task.ansible.out:changed: [smithi066.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-19T03:26:20.601 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-19T03:26:20.602 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010898", "end": "2020-11-19 03:26:20.577344", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-19 03:26:20.566446", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:20.717 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006085", "end": "2020-11-19 03:26:20.696982", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-19 03:26:20.690897", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:20.834 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005265", "end": "2020-11-19 03:26:20.807405", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-19 03:26:20.802140", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:20.982 INFO:teuthology.task.ansible.out:ok: [smithi154.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.035033", "end": "2020-11-19 03:26:20.949267", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-19 03:26:20.914234", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-19T03:26:21.022 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:21 +0000 (0:00:00.564) 0:01:40.069 ***** 2020-11-19T03:26:21.050 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-19T03:26:21.089 INFO:teuthology.task.ansible.out:Thursday 19 November 2020 03:26:21 +0000 (0:00:00.066) 0:01:40.136 ***** 2020-11-19T03:26:21.219 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-19T03:26:21.220 INFO:teuthology.task.ansible.out:changed: [smithi154.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-19T03:26:21.362 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-19T03:26:21.366 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-19T03:26:21.370 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-11-19T03:26:21.371 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi066.front.sepia.ceph.com : ok=108 changed=27 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 smithi154.front.sepia.ceph.com : ok=108 changed=27 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Thursday 19 November 2020 03:26:21 +0000 (0:00:00.281) 0:01:40.417 ***** =============================================================================== testnode : Ensure packages are not present. ----------------------------- 8.81s testnode : Install packages --------------------------------------------- 8.45s testnode : Zap all non-root disks --------------------------------------- 8.37s common : Update apt cache ----------------------------------------------- 2.88s testnode : Install apt keys --------------------------------------------- 2.51s testnode : Update apt cache. -------------------------------------------- 2.50s Gathering Facts --------------------------------------------------------- 1.81s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.66s testnode : Create logical volume(s) ------------------------------------- 1.53s users : Create all admin users with sudo access. ------------------------ 1.20s Gathering Facts --------------------------------------------------------- 0.87s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.73s testnode : Set the system's hostname ------------------------------------ 0.66s users : Write /keys-repo-sha1 sentinel file ----------------------------- 0.65s testnode : Make sure ntpd is running. ----------------------------------- 0.64s ansible-managed : Add authorized keys for the ansible user. ------------- 0.61s testnode : Install python-apt ------------------------------------------- 0.58s testnode : List any leftover Ceph artifacts from previous jobs ---------- 0.56s testnode : Write /scratch_devs --------------- 2020-11-19T03:26:21.373 INFO:teuthology.task.ansible.out:--------------------------- 0.51s testnode : Add ssh pubkeys ---------------------------------------------- 0.49s 2020-11-19T03:26:21.689 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi066.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2020-11-19T03:26:21.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:26:21.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:26:21.761 INFO:teuthology.orchestra.run.smithi066:> true 2020-11-19T03:26:21.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-19T03:26:21.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-19T03:26:22.033 INFO:teuthology.orchestra.run.smithi154:> true 2020-11-19T03:26:22.232 INFO:teuthology.run_tasks:Running task clock... 2020-11-19T03:26:22.259 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-19T03:26:22.261 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-19T03:26:22.262 INFO:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-19T03:26:22.342 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-19T03:26:22.343 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Command line: ntpd -gq 2020-11-19T03:26:22.344 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: proto: precision = 0.127 usec (-23) 2020-11-19T03:26:22.344 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T03:26:22.348 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: restrict ::: KOD does nothing without LIMITED. 2020-11-19T03:26:22.349 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Listen and drop on 0 v6wildcard [::]:123 2020-11-19T03:26:22.350 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-19T03:26:22.351 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Listen normally on 2 lo 127.0.0.1:123 2020-11-19T03:26:22.351 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Listen normally on 3 enp3s0f1 172.21.15.66:123 2020-11-19T03:26:22.354 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Listen normally on 4 lo [::1]:123 2020-11-19T03:26:22.356 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8bd%5]:123 2020-11-19T03:26:22.358 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:22 ntpd[5597]: Listening on routing socket on fd #22 for interface updates 2020-11-19T03:26:22.359 INFO:teuthology.orchestra.run.smithi066.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T03:26:22.360 INFO:teuthology.orchestra.run.smithi066.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-19T03:26:23.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:23.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:23.341 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:23.341 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:23.341 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:23.342 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:23.342 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:23.343 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:23 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:24.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:24 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:24.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:24 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:24.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:24 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:24.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:24 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:25.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:25.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:25.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:25.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:25.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:25.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:25.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:25.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:25 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:26.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:26 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:26.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:26 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:26.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:26 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:26.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:26 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:27.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:27.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:27.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:27.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:27.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:27.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:27.339 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:27.340 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:27.340 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:27.340 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:27.340 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:27.340 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:27 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:28.337 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:28 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:28.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:28 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:28.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:28 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:28.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:28 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:29.338 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:30.038 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:30.038 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:30.039 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:30.039 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:30.039 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:30.040 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:30.040 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:30.040 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-19T03:26:30.040 INFO:teuthology.orchestra.run.smithi066.stdout:19 Nov 03:26:29 ntpd[5597]: ntpd: time slew -0.004543 s 2020-11-19T03:26:30.041 INFO:teuthology.orchestra.run.smithi066.stdout:ntpd: time slew -0.004543s 2020-11-19T03:26:30.042 INFO:teuthology.orchestra.run.smithi066.stderr:19 Nov 03:26:29 ntpd[5597]: can't open /var/log/ntpstats/loopstats.20201119: Permission denied 2020-11-19T03:26:30.042 INFO:teuthology.orchestra.run.smithi066.stdout: remote refid st t when poll reach delay offset jitter 2020-11-19T03:26:30.043 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================== 2020-11-19T03:26:30.043 INFO:teuthology.orchestra.run.smithi066.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:30.043 INFO:teuthology.orchestra.run.smithi066.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:30.043 INFO:teuthology.orchestra.run.smithi066.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:30.044 INFO:teuthology.orchestra.run.smithi066.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:30.044 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-19T03:26:30.045 INFO:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-19T03:26:30.095 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-19T03:26:30.095 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Command line: ntpd -gq 2020-11-19T03:26:30.097 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: proto: precision = 0.082 usec (-23) 2020-11-19T03:26:30.097 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T03:26:30.098 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: restrict ::: KOD does nothing without LIMITED. 2020-11-19T03:26:30.098 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Listen and drop on 0 v6wildcard [::]:123 2020-11-19T03:26:30.098 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-19T03:26:30.098 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Listen normally on 2 lo 127.0.0.1:123 2020-11-19T03:26:30.099 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Listen normally on 3 enp3s0f1 172.21.15.154:123 2020-11-19T03:26:30.099 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Listen normally on 4 lo [::1]:123 2020-11-19T03:26:30.099 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7761%5]:123 2020-11-19T03:26:30.099 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:30 ntpd[5831]: Listening on routing socket on fd #22 for interface updates 2020-11-19T03:26:30.100 INFO:teuthology.orchestra.run.smithi154.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-19T03:26:30.100 INFO:teuthology.orchestra.run.smithi154.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-19T03:26:33.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:33 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:33.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:33 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:33.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:33 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:33.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:33 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:34.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:34.992 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:34.993 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:34.993 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:34.993 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:34.993 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:34.993 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:34.994 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:34 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:35.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:35.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:35.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:35.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:35.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:35.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:35.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:35.098 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:35 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:36.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:36.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:36.098 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:36.098 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:36.099 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:36.099 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:36.099 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:36.099 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:36 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:37.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:37.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:37.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:37.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:37.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:37.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:37.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:37.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:37 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:38.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:38.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:38.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:38.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:38.097 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:38.098 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:38.098 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:38.098 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:38 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:39.096 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:40.537 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:40.537 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:40.537 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:40.538 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-19T03:26:40.538 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: can't open /var/log/ntpstats/rawstats.20201119: Permission denied 2020-11-19T03:26:40.538 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-19T03:26:40.538 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: can't open /var/log/ntpstats/peerstats.20201119: Permission denied 2020-11-19T03:26:40.539 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-19T03:26:40.539 INFO:teuthology.orchestra.run.smithi154.stderr:19 Nov 03:26:39 ntpd[5831]: can't open /var/log/ntpstats/loopstats.20201119: Permission denied 2020-11-19T03:26:40.539 INFO:teuthology.orchestra.run.smithi154.stdout:19 Nov 03:26:39 ntpd[5831]: ntpd: time slew -0.007163 s 2020-11-19T03:26:40.540 INFO:teuthology.orchestra.run.smithi154.stdout:ntpd: time slew -0.007163s 2020-11-19T03:26:40.541 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2020-11-19T03:26:40.541 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2020-11-19T03:26:40.541 INFO:teuthology.orchestra.run.smithi154.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:40.542 INFO:teuthology.orchestra.run.smithi154.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:40.542 INFO:teuthology.orchestra.run.smithi154.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:40.542 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-19T03:26:40.542 INFO:teuthology.run_tasks:Running task install... 2020-11-19T03:26:40.552 DEBUG:teuthology.task.install:project ceph 2020-11-19T03:26:40.552 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}} 2020-11-19T03:26:40.553 DEBUG:teuthology.task.install:config {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'} 2020-11-19T03:26:40.553 INFO:teuthology.task.install:Using flavor: basic 2020-11-19T03:26:40.560 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-19T03:26:40.560 INFO:teuthology.task.install:extra packages: [] 2020-11-19T03:26:40.561 INFO:teuthology.orchestra.run.smithi066:> sudo lsb_release -is 2020-11-19T03:26:40.614 INFO:teuthology.orchestra.run.smithi066.stdout:Ubuntu 2020-11-19T03:26:40.619 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T03:26:40.619 INFO:teuthology.orchestra.run.smithi154:> sudo lsb_release -is 2020-11-19T03:26:40.620 INFO:teuthology.orchestra.run.smithi066:> sudo apt-key list | grep Ceph 2020-11-19T03:26:40.666 INFO:teuthology.orchestra.run.smithi154.stdout:Ubuntu 2020-11-19T03:26:40.671 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T03:26:40.672 INFO:teuthology.orchestra.run.smithi154:> sudo apt-key list | grep Ceph 2020-11-19T03:26:40.707 INFO:teuthology.orchestra.run.smithi066.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-19T03:26:40.758 INFO:teuthology.orchestra.run.smithi154.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-19T03:26:40.768 INFO:teuthology.orchestra.run.smithi066.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-19T03:26:40.769 INFO:teuthology.orchestra.run.smithi066.stdout:uid [ unknown] Ceph.com (release key) 2020-11-19T03:26:40.770 INFO:teuthology.orchestra.run.smithi066:> sudo lsb_release -is 2020-11-19T03:26:40.816 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-19T03:26:40.816 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph.com (release key) 2020-11-19T03:26:40.818 INFO:teuthology.orchestra.run.smithi154:> sudo lsb_release -is 2020-11-19T03:26:40.822 INFO:teuthology.orchestra.run.smithi066.stdout:Ubuntu 2020-11-19T03:26:40.827 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T03:26:40.827 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-19T03:26:40.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T03:26:40.872 INFO:teuthology.orchestra.run.smithi154.stdout:Ubuntu 2020-11-19T03:26:40.876 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T03:26:40.877 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-19T03:26:40.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T03:26:41.001 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default/ 2020-11-19T03:26:41.001 INFO:teuthology.task.install.deb:Package version is 16.0.0-7199-g409eac50-1bionic 2020-11-19T03:26:41.045 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default/ 2020-11-19T03:26:41.046 INFO:teuthology.task.install.deb:Package version is 16.0.0-7199-g409eac50-1bionic 2020-11-19T03:26:41.122 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:26:41.123 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T03:26:41.128 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T03:26:41.138 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:26:41.138 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:26:41.138 INFO:teuthology.orchestra.run.smithi066.stderr:155 bytes copied, 5.614e-05 s, 2.8 MB/s 2020-11-19T03:26:41.139 INFO:teuthology.orchestra.run.smithi066:> sudo apt-get update 2020-11-19T03:26:41.181 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:26:41.181 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T03:26:41.186 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-19T03:26:41.195 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:26:41.195 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:26:41.195 INFO:teuthology.orchestra.run.smithi154.stderr:155 bytes copied, 5.9941e-05 s, 2.6 MB/s 2020-11-19T03:26:41.196 INFO:teuthology.orchestra.run.smithi154:> sudo apt-get update 2020-11-19T03:26:41.374 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T03:26:41.434 INFO:teuthology.orchestra.run.smithi066.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic InRelease 2020-11-19T03:26:41.450 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T03:26:41.450 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-19T03:26:41.459 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-19T03:26:41.477 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic InRelease 2020-11-19T03:26:41.485 INFO:teuthology.orchestra.run.smithi066.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-19T03:26:41.486 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-19T03:26:41.502 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-19T03:26:41.510 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,941 B] 2020-11-19T03:26:41.527 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T03:26:41.528 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-19T03:26:41.552 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-19T03:26:41.552 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-19T03:26:41.570 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-19T03:26:41.594 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T03:26:41.595 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,941 B] 2020-11-19T03:26:41.642 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-19T03:26:41.943 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 26.7 kB in 1s (49.0 kB/s) 2020-11-19T03:26:42.052 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 26.7 kB in 1s (37.9 kB/s) 2020-11-19T03:26:43.255 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:26:43.304 INFO:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7199-g409eac50-1bionic cephadm=16.0.0-7199-g409eac50-1bionic ceph-mds=16.0.0-7199-g409eac50-1bionic ceph-mgr=16.0.0-7199-g409eac50-1bionic ceph-common=16.0.0-7199-g409eac50-1bionic ceph-fuse=16.0.0-7199-g409eac50-1bionic ceph-test=16.0.0-7199-g409eac50-1bionic radosgw=16.0.0-7199-g409eac50-1bionic python3-rados=16.0.0-7199-g409eac50-1bionic python3-rgw=16.0.0-7199-g409eac50-1bionic python3-cephfs=16.0.0-7199-g409eac50-1bionic python3-rbd=16.0.0-7199-g409eac50-1bionic libcephfs2=16.0.0-7199-g409eac50-1bionic libcephfs-dev=16.0.0-7199-g409eac50-1bionic librados2=16.0.0-7199-g409eac50-1bionic librbd1=16.0.0-7199-g409eac50-1bionic rbd-fuse=16.0.0-7199-g409eac50-1bionic 2020-11-19T03:26:43.330 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:26:43.374 INFO:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7199-g409eac50-1bionic cephadm=16.0.0-7199-g409eac50-1bionic ceph-mds=16.0.0-7199-g409eac50-1bionic ceph-mgr=16.0.0-7199-g409eac50-1bionic ceph-common=16.0.0-7199-g409eac50-1bionic ceph-fuse=16.0.0-7199-g409eac50-1bionic ceph-test=16.0.0-7199-g409eac50-1bionic radosgw=16.0.0-7199-g409eac50-1bionic python3-rados=16.0.0-7199-g409eac50-1bionic python3-rgw=16.0.0-7199-g409eac50-1bionic python3-cephfs=16.0.0-7199-g409eac50-1bionic python3-rbd=16.0.0-7199-g409eac50-1bionic libcephfs2=16.0.0-7199-g409eac50-1bionic libcephfs-dev=16.0.0-7199-g409eac50-1bionic librados2=16.0.0-7199-g409eac50-1bionic librbd1=16.0.0-7199-g409eac50-1bionic rbd-fuse=16.0.0-7199-g409eac50-1bionic 2020-11-19T03:26:43.393 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2020-11-19T03:26:43.459 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2020-11-19T03:26:43.656 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2020-11-19T03:26:44.573 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2020-11-19T03:26:44.574 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2020-11-19T03:26:44.574 INFO:teuthology.orchestra.run.smithi066.stdout:The following additional packages will be installed: 2020-11-19T03:26:44.575 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-19T03:26:44.575 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-19T03:26:44.576 INFO:teuthology.orchestra.run.smithi066.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-19T03:26:44.576 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-19T03:26:44.576 INFO:teuthology.orchestra.run.smithi066.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-19T03:26:44.577 INFO:teuthology.orchestra.run.smithi066.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-19T03:26:44.578 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-19T03:26:44.578 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-19T03:26:44.579 INFO:teuthology.orchestra.run.smithi066.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-19T03:26:44.579 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-19T03:26:44.579 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-19T03:26:44.580 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-19T03:26:44.581 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-19T03:26:44.582 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-19T03:26:44.582 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-19T03:26:44.582 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-19T03:26:44.583 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-19T03:26:44.583 INFO:teuthology.orchestra.run.smithi066.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T03:26:44.583 INFO:teuthology.orchestra.run.smithi066.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T03:26:44.584 INFO:teuthology.orchestra.run.smithi066.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T03:26:44.584 INFO:teuthology.orchestra.run.smithi066.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T03:26:44.584 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-19T03:26:44.586 INFO:teuthology.orchestra.run.smithi066.stdout:Suggested packages: 2020-11-19T03:26:44.586 INFO:teuthology.orchestra.run.smithi066.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-19T03:26:44.586 INFO:teuthology.orchestra.run.smithi066.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-19T03:26:44.587 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-19T03:26:44.587 INFO:teuthology.orchestra.run.smithi066.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-19T03:26:44.587 INFO:teuthology.orchestra.run.smithi066.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-19T03:26:44.588 INFO:teuthology.orchestra.run.smithi066.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-19T03:26:44.588 INFO:teuthology.orchestra.run.smithi066.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-19T03:26:44.589 INFO:teuthology.orchestra.run.smithi066.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-19T03:26:44.589 INFO:teuthology.orchestra.run.smithi066.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-19T03:26:44.590 INFO:teuthology.orchestra.run.smithi066.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-19T03:26:44.590 INFO:teuthology.orchestra.run.smithi066.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-19T03:26:44.590 INFO:teuthology.orchestra.run.smithi066.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-19T03:26:44.591 INFO:teuthology.orchestra.run.smithi066.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-19T03:26:44.591 INFO:teuthology.orchestra.run.smithi066.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-19T03:26:44.591 INFO:teuthology.orchestra.run.smithi066.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-19T03:26:44.591 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2020-11-19T03:26:44.592 INFO:teuthology.orchestra.run.smithi066.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-19T03:26:44.593 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-19T03:26:44.593 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-19T03:26:44.593 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-19T03:26:44.594 INFO:teuthology.orchestra.run.smithi066.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-19T03:26:44.595 INFO:teuthology.orchestra.run.smithi066.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-19T03:26:44.595 INFO:teuthology.orchestra.run.smithi066.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-19T03:26:44.595 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-19T03:26:44.596 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-19T03:26:44.596 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-19T03:26:44.596 INFO:teuthology.orchestra.run.smithi066.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-19T03:26:44.597 INFO:teuthology.orchestra.run.smithi066.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-19T03:26:44.597 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-19T03:26:44.597 INFO:teuthology.orchestra.run.smithi066.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-19T03:26:44.598 INFO:teuthology.orchestra.run.smithi066.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-19T03:26:44.598 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-19T03:26:44.598 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-19T03:26:44.599 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-19T03:26:44.600 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-19T03:26:44.600 INFO:teuthology.orchestra.run.smithi066.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T03:26:44.600 INFO:teuthology.orchestra.run.smithi066.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T03:26:44.601 INFO:teuthology.orchestra.run.smithi066.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T03:26:44.601 INFO:teuthology.orchestra.run.smithi066.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T03:26:44.601 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-19T03:26:44.601 INFO:teuthology.orchestra.run.smithi066.stdout:The following packages will be upgraded: 2020-11-19T03:26:44.602 INFO:teuthology.orchestra.run.smithi066.stdout: librados2 librbd1 2020-11-19T03:26:44.602 INFO:teuthology.orchestra.run.smithi066.stdout:2 upgraded, 110 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T03:26:44.602 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 168 MB of archives. 2020-11-19T03:26:44.603 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-19T03:26:44.603 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7199-g409eac50-1bionic [1,924 kB] 2020-11-19T03:26:44.603 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-19T03:26:44.604 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-19T03:26:44.604 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7199-g409eac50-1bionic [3,160 kB] 2020-11-19T03:26:44.604 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-19T03:26:44.605 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-19T03:26:44.605 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2020-11-19T03:26:44.605 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2020-11-19T03:26:44.606 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-19T03:26:44.606 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-19T03:26:44.606 INFO:teuthology.orchestra.run.smithi154.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-19T03:26:44.607 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-19T03:26:44.607 INFO:teuthology.orchestra.run.smithi154.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-19T03:26:44.607 INFO:teuthology.orchestra.run.smithi154.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-19T03:26:44.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-19T03:26:44.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-19T03:26:44.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-19T03:26:44.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-19T03:26:44.609 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-19T03:26:44.609 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-19T03:26:44.609 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-19T03:26:44.610 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-19T03:26:44.610 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-19T03:26:44.610 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-19T03:26:44.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-19T03:26:44.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T03:26:44.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T03:26:44.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T03:26:44.613 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T03:26:44.614 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-19T03:26:44.614 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2020-11-19T03:26:44.615 INFO:teuthology.orchestra.run.smithi154.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-19T03:26:44.615 INFO:teuthology.orchestra.run.smithi154.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-19T03:26:44.615 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-19T03:26:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-19T03:26:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-19T03:26:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-19T03:26:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-19T03:26:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-19T03:26:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-19T03:26:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-19T03:26:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-19T03:26:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-19T03:26:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-19T03:26:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-19T03:26:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-19T03:26:44.620 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2020-11-19T03:26:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-19T03:26:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-19T03:26:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-19T03:26:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-19T03:26:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-19T03:26:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-19T03:26:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-19T03:26:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-19T03:26:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-19T03:26:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-19T03:26:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-19T03:26:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-19T03:26:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-19T03:26:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-19T03:26:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-19T03:26:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-19T03:26:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-19T03:26:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-19T03:26:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-19T03:26:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-19T03:26:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-19T03:26:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-19T03:26:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-19T03:26:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-19T03:26:44.626 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages will be upgraded: 2020-11-19T03:26:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 librbd1 2020-11-19T03:26:44.627 INFO:teuthology.orchestra.run.smithi154.stdout:2 upgraded, 110 newly installed, 0 to remove and 13 not upgraded. 2020-11-19T03:26:44.627 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 168 MB of archives. 2020-11-19T03:26:44.627 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-19T03:26:44.628 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7199-g409eac50-1bionic [1,924 kB] 2020-11-19T03:26:44.628 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-19T03:26:44.628 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-19T03:26:44.628 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7199-g409eac50-1bionic [3,160 kB] 2020-11-19T03:26:44.628 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-19T03:26:44.629 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-19T03:26:44.629 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-19T03:26:44.629 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7199-g409eac50-1bionic [478 kB] 2020-11-19T03:26:44.629 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7199-g409eac50-1bionic [332 kB] 2020-11-19T03:26:44.629 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7199-g409eac50-1bionic [20.6 kB] 2020-11-19T03:26:44.630 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7199-g409eac50-1bionic [168 kB] 2020-11-19T03:26:44.630 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7199-g409eac50-1bionic [27.7 kB] 2020-11-19T03:26:44.630 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7199-g409eac50-1bionic [330 kB] 2020-11-19T03:26:44.630 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7199-g409eac50-1bionic [3,178 kB] 2020-11-19T03:26:44.630 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-19T03:26:44.631 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-19T03:26:44.632 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-19T03:26:44.632 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7199-g409eac50-1bionic [478 kB] 2020-11-19T03:26:44.632 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-19T03:26:44.633 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-19T03:26:44.633 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-19T03:26:44.633 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-19T03:26:44.640 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-19T03:26:44.668 INFO:teuthology.orchestra.run.smithi066.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7199-g409eac50-1bionic [332 kB] 2020-11-19T03:26:44.669 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-19T03:26:44.672 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-19T03:26:44.676 INFO:teuthology.orchestra.run.smithi066.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-19T03:26:44.680 INFO:teuthology.orchestra.run.smithi066.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-19T03:26:44.692 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-19T03:26:44.700 INFO:teuthology.orchestra.run.smithi066.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7199-g409eac50-1bionic [20.6 kB] 2020-11-19T03:26:44.701 INFO:teuthology.orchestra.run.smithi066.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7199-g409eac50-1bionic [168 kB] 2020-11-19T03:26:44.703 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-19T03:26:44.704 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-19T03:26:44.708 INFO:teuthology.orchestra.run.smithi066.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7199-g409eac50-1bionic [27.7 kB] 2020-11-19T03:26:44.708 INFO:teuthology.orchestra.run.smithi066.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-19T03:26:44.709 INFO:teuthology.orchestra.run.smithi066.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7199-g409eac50-1bionic [330 kB] 2020-11-19T03:26:44.716 INFO:teuthology.orchestra.run.smithi066.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-19T03:26:44.724 INFO:teuthology.orchestra.run.smithi066.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-19T03:26:44.733 INFO:teuthology.orchestra.run.smithi066.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-19T03:26:44.741 INFO:teuthology.orchestra.run.smithi066.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-19T03:26:44.741 INFO:teuthology.orchestra.run.smithi066.stdout:Get:21 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7199-g409eac50-1bionic [3,178 kB] 2020-11-19T03:26:44.760 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7199-g409eac50-1bionic [99.2 kB] 2020-11-19T03:26:44.763 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7199-g409eac50-1bionic [414 kB] 2020-11-19T03:26:44.767 INFO:teuthology.orchestra.run.smithi066.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-19T03:26:44.771 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-19T03:26:44.772 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-19T03:26:44.772 INFO:teuthology.orchestra.run.smithi066.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-19T03:26:44.774 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-19T03:26:44.774 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-19T03:26:44.775 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-19T03:26:44.776 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7199-g409eac50-1bionic [20.5 MB] 2020-11-19T03:26:44.778 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-19T03:26:44.780 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-19T03:26:44.781 INFO:teuthology.orchestra.run.smithi066.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-19T03:26:44.795 INFO:teuthology.orchestra.run.smithi066.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-19T03:26:44.850 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-19T03:26:44.851 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-19T03:26:44.861 INFO:teuthology.orchestra.run.smithi066.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-19T03:26:44.862 INFO:teuthology.orchestra.run.smithi066.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-19T03:26:44.862 INFO:teuthology.orchestra.run.smithi066.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-19T03:26:44.864 INFO:teuthology.orchestra.run.smithi066.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-19T03:26:44.865 INFO:teuthology.orchestra.run.smithi066.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-19T03:26:44.865 INFO:teuthology.orchestra.run.smithi066.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-19T03:26:44.868 INFO:teuthology.orchestra.run.smithi066.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-19T03:26:44.871 INFO:teuthology.orchestra.run.smithi066.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-19T03:26:44.872 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-19T03:26:44.929 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-19T03:26:44.930 INFO:teuthology.orchestra.run.smithi154.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-19T03:26:44.931 INFO:teuthology.orchestra.run.smithi154.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-19T03:26:44.935 INFO:teuthology.orchestra.run.smithi154.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-19T03:26:44.940 INFO:teuthology.orchestra.run.smithi066.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-19T03:26:44.940 INFO:teuthology.orchestra.run.smithi066.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-19T03:26:44.943 INFO:teuthology.orchestra.run.smithi154.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-19T03:26:44.949 INFO:teuthology.orchestra.run.smithi154.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-19T03:26:44.949 INFO:teuthology.orchestra.run.smithi154.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-19T03:26:44.984 INFO:teuthology.orchestra.run.smithi066.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7199-g409eac50-1bionic [99.2 kB] 2020-11-19T03:26:44.987 INFO:teuthology.orchestra.run.smithi066.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7199-g409eac50-1bionic [414 kB] 2020-11-19T03:26:45.002 INFO:teuthology.orchestra.run.smithi066.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7199-g409eac50-1bionic [20.5 MB] 2020-11-19T03:26:45.009 INFO:teuthology.orchestra.run.smithi154.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-19T03:26:45.019 INFO:teuthology.orchestra.run.smithi066.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-19T03:26:45.019 INFO:teuthology.orchestra.run.smithi066.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-19T03:26:45.020 INFO:teuthology.orchestra.run.smithi066.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-19T03:26:45.021 INFO:teuthology.orchestra.run.smithi066.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-19T03:26:45.024 INFO:teuthology.orchestra.run.smithi066.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-19T03:26:45.032 INFO:teuthology.orchestra.run.smithi066.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-19T03:26:45.036 INFO:teuthology.orchestra.run.smithi066.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-19T03:26:45.036 INFO:teuthology.orchestra.run.smithi066.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-19T03:26:45.098 INFO:teuthology.orchestra.run.smithi066.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-19T03:26:45.101 INFO:teuthology.orchestra.run.smithi154.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-19T03:26:45.101 INFO:teuthology.orchestra.run.smithi154.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-19T03:26:45.181 INFO:teuthology.orchestra.run.smithi066.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-19T03:26:45.182 INFO:teuthology.orchestra.run.smithi066.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-19T03:26:45.379 INFO:teuthology.orchestra.run.smithi154.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-19T03:26:45.434 INFO:teuthology.orchestra.run.smithi154.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-19T03:26:45.436 INFO:teuthology.orchestra.run.smithi154.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-19T03:26:45.460 INFO:teuthology.orchestra.run.smithi154.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-19T03:26:45.460 INFO:teuthology.orchestra.run.smithi154.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-19T03:26:45.461 INFO:teuthology.orchestra.run.smithi154.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-19T03:26:45.461 INFO:teuthology.orchestra.run.smithi154.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-19T03:26:45.462 INFO:teuthology.orchestra.run.smithi154.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-19T03:26:45.462 INFO:teuthology.orchestra.run.smithi154.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-19T03:26:45.463 INFO:teuthology.orchestra.run.smithi154.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-19T03:26:45.466 INFO:teuthology.orchestra.run.smithi066.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-19T03:26:45.486 INFO:teuthology.orchestra.run.smithi066.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-19T03:26:45.505 INFO:teuthology.orchestra.run.smithi066.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-19T03:26:45.539 INFO:teuthology.orchestra.run.smithi154.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-19T03:26:45.540 INFO:teuthology.orchestra.run.smithi154.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-19T03:26:45.541 INFO:teuthology.orchestra.run.smithi066.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-19T03:26:45.542 INFO:teuthology.orchestra.run.smithi066.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-19T03:26:45.542 INFO:teuthology.orchestra.run.smithi066.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-19T03:26:45.543 INFO:teuthology.orchestra.run.smithi066.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi066.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-19T03:26:45.545 INFO:teuthology.orchestra.run.smithi066.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-19T03:26:45.547 INFO:teuthology.orchestra.run.smithi066.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-19T03:26:45.618 INFO:teuthology.orchestra.run.smithi154.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-19T03:26:45.623 INFO:teuthology.orchestra.run.smithi154.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-19T03:26:45.626 INFO:teuthology.orchestra.run.smithi154.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-19T03:26:45.626 INFO:teuthology.orchestra.run.smithi154.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-19T03:26:45.627 INFO:teuthology.orchestra.run.smithi154.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-19T03:26:45.627 INFO:teuthology.orchestra.run.smithi066.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-19T03:26:45.628 INFO:teuthology.orchestra.run.smithi154.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-19T03:26:45.629 INFO:teuthology.orchestra.run.smithi066.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-19T03:26:45.629 INFO:teuthology.orchestra.run.smithi154.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-19T03:26:45.630 INFO:teuthology.orchestra.run.smithi154.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-19T03:26:45.636 INFO:teuthology.orchestra.run.smithi066.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-19T03:26:45.637 INFO:teuthology.orchestra.run.smithi066.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-19T03:26:45.640 INFO:teuthology.orchestra.run.smithi066.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-19T03:26:45.641 INFO:teuthology.orchestra.run.smithi066.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-19T03:26:45.641 INFO:teuthology.orchestra.run.smithi066.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-19T03:26:45.641 INFO:teuthology.orchestra.run.smithi066.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-19T03:26:45.643 INFO:teuthology.orchestra.run.smithi066.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-19T03:26:45.697 INFO:teuthology.orchestra.run.smithi154.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-19T03:26:45.703 INFO:teuthology.orchestra.run.smithi154.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-19T03:26:45.706 INFO:teuthology.orchestra.run.smithi066.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-19T03:26:45.723 INFO:teuthology.orchestra.run.smithi066.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-19T03:26:45.726 INFO:teuthology.orchestra.run.smithi066.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-19T03:26:45.735 INFO:teuthology.orchestra.run.smithi066.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-19T03:26:45.741 INFO:teuthology.orchestra.run.smithi066.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-19T03:26:45.742 INFO:teuthology.orchestra.run.smithi066.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-19T03:26:45.743 INFO:teuthology.orchestra.run.smithi066.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-19T03:26:45.743 INFO:teuthology.orchestra.run.smithi066.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-19T03:26:45.744 INFO:teuthology.orchestra.run.smithi066.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-19T03:26:45.777 INFO:teuthology.orchestra.run.smithi154.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-19T03:26:45.784 INFO:teuthology.orchestra.run.smithi154.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-19T03:26:45.784 INFO:teuthology.orchestra.run.smithi154.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-19T03:26:45.785 INFO:teuthology.orchestra.run.smithi066.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-19T03:26:45.785 INFO:teuthology.orchestra.run.smithi154.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-19T03:26:45.786 INFO:teuthology.orchestra.run.smithi154.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-19T03:26:45.787 INFO:teuthology.orchestra.run.smithi154.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-19T03:26:45.790 INFO:teuthology.orchestra.run.smithi154.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-19T03:26:45.790 INFO:teuthology.orchestra.run.smithi154.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-19T03:26:45.826 INFO:teuthology.orchestra.run.smithi066.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-19T03:26:45.827 INFO:teuthology.orchestra.run.smithi066.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-19T03:26:45.828 INFO:teuthology.orchestra.run.smithi066.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-19T03:26:45.830 INFO:teuthology.orchestra.run.smithi066.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-19T03:26:45.852 INFO:teuthology.orchestra.run.smithi066.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-19T03:26:45.853 INFO:teuthology.orchestra.run.smithi066.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-19T03:26:45.853 INFO:teuthology.orchestra.run.smithi066.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-19T03:26:45.856 INFO:teuthology.orchestra.run.smithi154.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-19T03:26:45.856 INFO:teuthology.orchestra.run.smithi154.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-19T03:26:45.936 INFO:teuthology.orchestra.run.smithi154.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-19T03:26:45.947 INFO:teuthology.orchestra.run.smithi066.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-19T03:26:45.947 INFO:teuthology.orchestra.run.smithi066.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-19T03:26:45.948 INFO:teuthology.orchestra.run.smithi066.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-19T03:26:45.952 INFO:teuthology.orchestra.run.smithi066.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-19T03:26:45.953 INFO:teuthology.orchestra.run.smithi066.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-19T03:26:45.953 INFO:teuthology.orchestra.run.smithi066.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-19T03:26:45.954 INFO:teuthology.orchestra.run.smithi066.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-19T03:26:45.955 INFO:teuthology.orchestra.run.smithi066.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-19T03:26:45.958 INFO:teuthology.orchestra.run.smithi154.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-19T03:26:45.958 INFO:teuthology.orchestra.run.smithi154.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-19T03:26:45.959 INFO:teuthology.orchestra.run.smithi154.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-19T03:26:45.986 INFO:teuthology.orchestra.run.smithi066.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-19T03:26:46.002 INFO:teuthology.orchestra.run.smithi154.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7199-g409eac50-1bionic [5,564 kB] 2020-11-19T03:26:46.086 INFO:teuthology.orchestra.run.smithi154.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-19T03:26:46.087 INFO:teuthology.orchestra.run.smithi154.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-19T03:26:46.087 INFO:teuthology.orchestra.run.smithi154.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-19T03:26:46.094 INFO:teuthology.orchestra.run.smithi154.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-19T03:26:46.095 INFO:teuthology.orchestra.run.smithi154.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-19T03:26:46.095 INFO:teuthology.orchestra.run.smithi154.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-19T03:26:46.096 INFO:teuthology.orchestra.run.smithi154.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-19T03:26:46.098 INFO:teuthology.orchestra.run.smithi154.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-19T03:26:46.098 INFO:teuthology.orchestra.run.smithi154.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-19T03:26:46.339 INFO:teuthology.orchestra.run.smithi154.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7199-g409eac50-1bionic [165 kB] 2020-11-19T03:26:46.363 INFO:teuthology.orchestra.run.smithi154.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7199-g409eac50-1bionic [1,361 kB] 2020-11-19T03:26:46.435 INFO:teuthology.orchestra.run.smithi154.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7199-g409eac50-1bionic [6,661 kB] 2020-11-19T03:26:46.495 INFO:teuthology.orchestra.run.smithi066.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7199-g409eac50-1bionic [5,564 kB] 2020-11-19T03:26:46.839 INFO:teuthology.orchestra.run.smithi154.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7199-g409eac50-1bionic [24.6 MB] 2020-11-19T03:26:46.895 INFO:teuthology.orchestra.run.smithi066.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7199-g409eac50-1bionic [165 kB] 2020-11-19T03:26:46.938 INFO:teuthology.orchestra.run.smithi066.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7199-g409eac50-1bionic [1,361 kB] 2020-11-19T03:26:47.021 INFO:teuthology.orchestra.run.smithi066.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7199-g409eac50-1bionic [6,661 kB] 2020-11-19T03:26:47.515 INFO:teuthology.orchestra.run.smithi066.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7199-g409eac50-1bionic [24.6 MB] 2020-11-19T03:26:48.314 INFO:teuthology.orchestra.run.smithi154.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7199-g409eac50-1bionic [3,816 B] 2020-11-19T03:26:48.314 INFO:teuthology.orchestra.run.smithi154.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7199-g409eac50-1bionic [563 kB] 2020-11-19T03:26:48.327 INFO:teuthology.orchestra.run.smithi154.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7199-g409eac50-1bionic [1,941 kB] 2020-11-19T03:26:48.461 INFO:teuthology.orchestra.run.smithi154.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7199-g409eac50-1bionic [51.7 kB] 2020-11-19T03:26:49.624 INFO:teuthology.orchestra.run.smithi066.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7199-g409eac50-1bionic [3,816 B] 2020-11-19T03:26:49.624 INFO:teuthology.orchestra.run.smithi066.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7199-g409eac50-1bionic [563 kB] 2020-11-19T03:26:49.624 INFO:teuthology.orchestra.run.smithi066.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7199-g409eac50-1bionic [1,941 kB] 2020-11-19T03:26:49.624 INFO:teuthology.orchestra.run.smithi066.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7199-g409eac50-1bionic [51.7 kB] 2020-11-19T03:26:49.625 INFO:teuthology.orchestra.run.smithi066.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7199-g409eac50-1bionic [56.2 kB] 2020-11-19T03:26:49.625 INFO:teuthology.orchestra.run.smithi066.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7199-g409eac50-1bionic [2,877 kB] 2020-11-19T03:26:49.625 INFO:teuthology.orchestra.run.smithi154.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7199-g409eac50-1bionic [56.2 kB] 2020-11-19T03:26:49.625 INFO:teuthology.orchestra.run.smithi154.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7199-g409eac50-1bionic [2,877 kB] 2020-11-19T03:26:49.626 INFO:teuthology.orchestra.run.smithi154.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7199-g409eac50-1bionic [7,885 kB] 2020-11-19T03:26:49.626 INFO:teuthology.orchestra.run.smithi154.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7199-g409eac50-1bionic [3,932 B] 2020-11-19T03:26:49.626 INFO:teuthology.orchestra.run.smithi154.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7199-g409eac50-1bionic [22.3 kB] 2020-11-19T03:26:49.626 INFO:teuthology.orchestra.run.smithi154.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7199-g409eac50-1bionic [49.6 MB] 2020-11-19T03:26:49.710 INFO:teuthology.orchestra.run.smithi066.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7199-g409eac50-1bionic [7,885 kB] 2020-11-19T03:26:50.198 INFO:teuthology.orchestra.run.smithi066.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7199-g409eac50-1bionic [3,932 B] 2020-11-19T03:26:50.199 INFO:teuthology.orchestra.run.smithi066.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7199-g409eac50-1bionic [22.3 kB] 2020-11-19T03:26:50.199 INFO:teuthology.orchestra.run.smithi066.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7199-g409eac50-1bionic [49.6 MB] 2020-11-19T03:26:52.357 INFO:teuthology.orchestra.run.smithi154.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7199-g409eac50-1bionic [19.5 kB] 2020-11-19T03:26:52.357 INFO:teuthology.orchestra.run.smithi154.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7199-g409eac50-1bionic [9,854 kB] 2020-11-19T03:26:52.933 INFO:teuthology.orchestra.run.smithi154.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7199-g409eac50-1bionic [76.9 kB] 2020-11-19T03:26:53.220 INFO:teuthology.orchestra.run.smithi066.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7199-g409eac50-1bionic [19.5 kB] 2020-11-19T03:26:53.220 INFO:teuthology.orchestra.run.smithi066.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7199-g409eac50-1bionic [9,854 kB] 2020-11-19T03:26:53.538 INFO:teuthology.orchestra.run.smithi066.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuval-fix-47904/409eac507e130adcc9fee71aa5fd16ed7f289313/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7199-g409eac50-1bionic [76.9 kB] 2020-11-19T03:26:54.026 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 168 MB in 9s (18.7 MB/s) 2020-11-19T03:26:54.026 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-19T03:26:54.026 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137607 files and directories currently installed.) 2020-11-19T03:26:54.027 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-19T03:26:54.028 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T03:26:54.120 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-19T03:26:54.136 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-19T03:26:54.221 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T03:26:54.265 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 168 MB in 9s (17.7 MB/s) 2020-11-19T03:26:54.433 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-19T03:26:54.482 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137607 files and directories currently installed.) 2020-11-19T03:26:54.486 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-19T03:26:54.523 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T03:26:54.539 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../002-librbd1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:54.666 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librbd1 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T03:26:54.851 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-19T03:26:54.866 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-19T03:26:54.951 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T03:26:55.209 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../003-librados2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:55.326 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../002-librbd1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:55.337 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librados2 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T03:26:55.422 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking librbd1 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T03:26:55.974 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs2. 2020-11-19T03:26:55.983 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../003-librados2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:55.989 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:56.032 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:56.126 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking librados2 (16.0.0-7199-g409eac50-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-19T03:26:56.327 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rados. 2020-11-19T03:26:56.342 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:56.376 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:56.713 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-19T03:26:56.728 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:26:56.738 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libcephfs2. 2020-11-19T03:26:56.753 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:56.763 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:56.806 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:57.015 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cephfs. 2020-11-19T03:26:57.029 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:57.065 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:57.099 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-rados. 2020-11-19T03:26:57.113 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:57.282 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:57.418 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-19T03:26:57.433 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:26:57.476 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:57.677 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-19T03:26:57.691 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:26:57.735 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:57.788 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-prettytable. 2020-11-19T03:26:57.804 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-19T03:26:57.846 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-19T03:26:58.005 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-cephfs. 2020-11-19T03:26:58.020 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:58.055 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:58.066 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rbd. 2020-11-19T03:26:58.081 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:58.116 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:58.374 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-19T03:26:58.389 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:26:58.432 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:58.493 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-19T03:26:58.507 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-19T03:26:58.543 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T03:26:59.414 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-prettytable. 2020-11-19T03:26:59.415 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-19T03:26:59.415 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-19T03:26:59.415 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-rbd. 2020-11-19T03:26:59.416 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:59.416 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:59.417 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-19T03:26:59.417 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-19T03:26:59.417 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T03:26:59.418 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librgw2. 2020-11-19T03:26:59.418 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../013-librgw2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:59.419 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:26:59.432 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-19T03:26:59.447 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-19T03:26:59.491 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T03:26:59.860 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-19T03:26:59.868 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rgw. 2020-11-19T03:26:59.874 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-19T03:26:59.882 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:26:59.927 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:00.060 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T03:27:00.288 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-19T03:27:00.302 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-19T03:27:00.346 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T03:27:00.396 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package librgw2. 2020-11-19T03:27:00.411 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../013-librgw2_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:00.456 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:00.641 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-19T03:27:00.657 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-19T03:27:00.699 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T03:27:00.982 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-rgw. 2020-11-19T03:27:00.996 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:01.032 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:01.052 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liboath0. 2020-11-19T03:27:01.066 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-19T03:27:01.110 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-19T03:27:01.351 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-19T03:27:01.364 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-19T03:27:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T03:27:01.422 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libradosstriper1. 2020-11-19T03:27:01.438 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:01.480 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:01.720 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-19T03:27:01.736 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-19T03:27:01.779 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T03:27:01.933 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-common. 2020-11-19T03:27:01.949 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:01.983 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:02.149 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package liboath0. 2020-11-19T03:27:02.162 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-19T03:27:02.198 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-19T03:27:02.518 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libradosstriper1. 2020-11-19T03:27:02.532 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:02.568 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:03.055 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-common. 2020-11-19T03:27:03.070 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:03.113 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:03.872 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-base. 2020-11-19T03:27:04.037 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:04.037 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:04.919 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-dateutil. 2020-11-19T03:27:04.934 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-19T03:27:04.968 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-19T03:27:04.987 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-base. 2020-11-19T03:27:05.001 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:05.046 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:05.305 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-19T03:27:05.319 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-19T03:27:05.363 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-19T03:27:05.649 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-19T03:27:05.664 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-19T03:27:05.708 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-19T03:27:06.036 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cryptography. 2020-11-19T03:27:06.049 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-19T03:27:06.094 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T03:27:06.183 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-dateutil. 2020-11-19T03:27:06.198 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-19T03:27:06.242 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-19T03:27:06.523 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-openssl. 2020-11-19T03:27:06.538 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-19T03:27:06.574 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T03:27:06.611 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-19T03:27:06.624 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-19T03:27:06.661 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-19T03:27:06.825 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-19T03:27:06.840 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:06.883 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:06.897 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-19T03:27:06.910 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-19T03:27:06.939 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-19T03:27:07.255 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-19T03:27:07.269 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-19T03:27:07.300 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-cryptography. 2020-11-19T03:27:07.314 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-19T03:27:07.405 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T03:27:07.563 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-19T03:27:07.866 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-19T03:27:07.881 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-19T03:27:07.924 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-19T03:27:07.928 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-openssl. 2020-11-19T03:27:07.937 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-19T03:27:07.978 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T03:27:08.261 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jwt. 2020-11-19T03:27:08.264 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-19T03:27:08.278 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-19T03:27:08.278 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:08.314 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:08.321 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-19T03:27:08.651 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-logutils. 2020-11-19T03:27:10.655 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-19T03:27:10.655 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-19T03:27:10.655 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-19T03:27:10.655 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-19T03:27:10.656 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-19T03:27:10.656 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-19T03:27:10.656 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-jwt. 2020-11-19T03:27:10.656 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-19T03:27:10.657 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-19T03:27:10.657 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-logutils. 2020-11-19T03:27:10.657 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-19T03:27:10.657 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-19T03:27:10.657 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-19T03:27:10.658 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-19T03:27:10.658 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-19T03:27:10.658 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-mako. 2020-11-19T03:27:10.658 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-19T03:27:10.658 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-19T03:27:10.659 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-19T03:27:10.659 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-19T03:27:10.659 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-19T03:27:10.659 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-19T03:27:10.659 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-mako. 2020-11-19T03:27:10.660 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-19T03:27:10.660 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-19T03:27:10.660 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-19T03:27:10.660 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-19T03:27:10.660 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-19T03:27:10.661 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-19T03:27:10.661 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-19T03:27:10.661 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-19T03:27:10.661 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webob. 2020-11-19T03:27:10.661 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-19T03:27:10.662 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T03:27:10.662 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-19T03:27:10.899 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-bs4. 2020-11-19T03:27:10.915 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-19T03:27:10.944 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-19T03:27:10.957 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-19T03:27:10.958 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-19T03:27:10.986 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-19T03:27:11.282 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-waitress. 2020-11-19T03:27:11.290 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-19T03:27:11.305 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-19T03:27:11.320 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-19T03:27:11.335 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-19T03:27:11.364 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-19T03:27:11.658 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-webob. 2020-11-19T03:27:11.672 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-19T03:27:11.680 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tempita. 2020-11-19T03:27:11.693 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-19T03:27:11.716 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T03:27:11.738 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-19T03:27:12.041 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-paste. 2020-11-19T03:27:12.044 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-bs4. 2020-11-19T03:27:12.054 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-19T03:27:12.059 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-19T03:27:12.091 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T03:27:12.103 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-19T03:27:12.389 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-waitress. 2020-11-19T03:27:12.405 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-19T03:27:12.427 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-19T03:27:12.439 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-19T03:27:12.442 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-19T03:27:12.485 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T03:27:12.771 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-tempita. 2020-11-19T03:27:12.787 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-19T03:27:12.813 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-19T03:27:12.825 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-19T03:27:12.828 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-19T03:27:12.871 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-19T03:27:13.111 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-paste. 2020-11-19T03:27:13.126 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-19T03:27:13.170 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T03:27:13.199 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webtest. 2020-11-19T03:27:13.214 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-19T03:27:13.266 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T03:27:13.489 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-19T03:27:15.436 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pecan. 2020-11-19T03:27:15.436 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-19T03:27:15.437 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-19T03:27:15.437 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-19T03:27:15.437 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-19T03:27:15.437 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T03:27:15.437 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr. 2020-11-19T03:27:15.438 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:15.438 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:15.438 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mon. 2020-11-19T03:27:15.438 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:15.438 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:15.439 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-19T03:27:15.439 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T03:27:15.439 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-19T03:27:15.439 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-19T03:27:15.440 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-19T03:27:15.440 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-webtest. 2020-11-19T03:27:15.440 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-19T03:27:15.440 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T03:27:15.440 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pecan. 2020-11-19T03:27:15.441 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-19T03:27:15.441 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-19T03:27:15.441 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-19T03:27:15.441 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-19T03:27:15.441 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T03:27:15.480 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mgr. 2020-11-19T03:27:15.495 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:15.539 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:15.950 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-osd. 2020-11-19T03:27:15.964 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:16.008 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:16.050 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mon. 2020-11-19T03:27:16.065 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:16.109 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:17.098 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-osd. 2020-11-19T03:27:17.111 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:17.154 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:18.621 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph. 2020-11-19T03:27:18.935 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../048-ceph_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:18.935 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:19.015 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-fuse. 2020-11-19T03:27:19.029 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:19.074 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:19.219 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph. 2020-11-19T03:27:19.232 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../048-ceph_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:19.282 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:19.460 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mds. 2020-11-19T03:27:19.474 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:19.510 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:19.605 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-fuse. 2020-11-19T03:27:19.619 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:19.654 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:20.038 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cephadm. 2020-11-19T03:27:20.053 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../051-cephadm_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:20.065 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mds. 2020-11-19T03:27:20.079 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:20.097 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:20.123 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:20.424 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jinja2. 2020-11-19T03:27:20.439 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-19T03:27:20.483 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T03:27:20.660 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package cephadm. 2020-11-19T03:27:20.675 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../051-cephadm_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:20.718 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:20.794 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-19T03:27:20.809 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:20.852 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:21.071 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-jinja2. 2020-11-19T03:27:21.084 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-19T03:27:21.122 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T03:27:21.206 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-19T03:27:21.220 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-19T03:27:21.264 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-19T03:27:21.424 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-19T03:27:21.440 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:21.482 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:21.576 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-routes. 2020-11-19T03:27:21.589 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-19T03:27:21.633 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-19T03:27:21.852 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-19T03:27:21.868 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-19T03:27:21.911 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-19T03:27:21.944 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-19T03:27:21.958 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:22.020 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:22.247 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-routes. 2020-11-19T03:27:22.260 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-19T03:27:22.297 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-19T03:27:22.624 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-19T03:27:22.639 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:22.682 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:22.885 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-numpy. 2020-11-19T03:27:22.903 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-19T03:27:22.944 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T03:27:23.558 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-decorator. 2020-11-19T03:27:23.573 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-19T03:27:23.599 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-numpy. 2020-11-19T03:27:25.181 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-19T03:27:25.181 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-scipy. 2020-11-19T03:27:25.181 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-19T03:27:25.181 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T03:27:25.182 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-19T03:27:25.182 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T03:27:25.182 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-decorator. 2020-11-19T03:27:25.182 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-19T03:27:25.183 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-19T03:27:25.183 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-scipy. 2020-11-19T03:27:25.183 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-19T03:27:25.183 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T03:27:25.269 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-19T03:27:25.285 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-19T03:27:25.327 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-19T03:27:25.773 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-joblib. 2020-11-19T03:27:25.789 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-19T03:27:25.831 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-19T03:27:25.942 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-19T03:27:25.956 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-19T03:27:26.001 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-19T03:27:26.201 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn. 2020-11-19T03:27:26.214 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-19T03:27:26.269 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-19T03:27:26.495 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-joblib. 2020-11-19T03:27:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-19T03:27:26.554 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-19T03:27:26.815 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-19T03:27:26.829 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:26.874 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:26.907 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-sklearn. 2020-11-19T03:27:26.920 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-19T03:27:26.965 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-19T03:27:27.563 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-19T03:27:27.580 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:27.621 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:28.035 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-monotonic. 2020-11-19T03:27:28.050 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-19T03:27:28.093 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-19T03:27:28.421 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-fasteners. 2020-11-19T03:27:28.438 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-19T03:27:28.480 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-19T03:27:28.801 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-monotonic. 2020-11-19T03:27:29.610 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-httplib2. 2020-11-19T03:27:29.612 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-19T03:27:29.612 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-19T03:27:29.613 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-fasteners. 2020-11-19T03:27:29.613 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-19T03:27:29.613 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-19T03:27:29.613 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-httplib2. 2020-11-19T03:27:29.614 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-19T03:27:29.614 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-19T03:27:29.614 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T03:27:29.614 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-19T03:27:29.614 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-19T03:27:29.615 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-19T03:27:29.615 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-19T03:27:29.615 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-19T03:27:29.616 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T03:27:29.621 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T03:27:29.916 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-19T03:27:29.931 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-19T03:27:29.941 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rsa. 2020-11-19T03:27:29.958 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-19T03:27:29.974 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-19T03:27:29.999 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-19T03:27:30.336 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-19T03:27:30.350 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-19T03:27:30.352 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-19T03:27:30.368 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-19T03:27:30.394 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T03:27:30.411 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T03:27:30.706 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-rsa. 2020-11-19T03:27:30.720 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-19T03:27:30.764 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-19T03:27:30.857 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-19T03:27:30.873 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-19T03:27:30.915 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-19T03:27:31.075 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-19T03:27:31.089 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-19T03:27:31.142 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T03:27:31.276 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-setuptools. 2020-11-19T03:27:31.293 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-19T03:27:31.335 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-19T03:27:31.547 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-19T03:27:31.561 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-19T03:27:31.596 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-19T03:27:31.730 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-websocket. 2020-11-19T03:27:31.743 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-19T03:27:31.788 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T03:27:31.941 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-setuptools. 2020-11-19T03:27:31.958 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-19T03:27:31.999 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-19T03:27:32.100 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-19T03:27:32.114 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-19T03:27:32.158 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T03:27:32.328 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-websocket. 2020-11-19T03:27:32.341 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-19T03:27:32.386 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T03:27:32.578 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-19T03:27:32.592 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:32.636 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:32.739 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-19T03:27:32.755 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-19T03:27:32.797 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T03:27:32.973 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-19T03:27:32.989 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-19T03:27:33.031 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-19T03:27:33.210 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-19T03:27:33.226 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:33.276 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:33.392 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-19T03:27:33.410 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-19T03:27:33.454 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T03:27:33.613 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-19T03:27:35.168 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-19T03:27:35.168 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:35.168 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:35.169 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-19T03:27:35.169 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-19T03:27:35.169 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-19T03:27:35.170 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-19T03:27:35.170 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-19T03:27:35.170 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T03:27:35.170 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package jq. 2020-11-19T03:27:35.171 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-19T03:27:35.171 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-19T03:27:35.171 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-19T03:27:35.171 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-19T03:27:35.172 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-19T03:27:35.172 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-19T03:27:35.172 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T03:27:35.172 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-19T03:27:35.172 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7199-g409eac50-1bionic_all.deb ... 2020-11-19T03:27:35.173 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:35.173 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-19T03:27:35.173 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-19T03:27:35.173 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-19T03:27:35.232 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-19T03:27:35.248 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-19T03:27:35.350 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package socat. 2020-11-19T03:27:35.365 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-19T03:27:35.368 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T03:27:35.408 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-19T03:27:35.663 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package jq. 2020-11-19T03:27:35.677 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-19T03:27:35.721 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-19T03:27:35.795 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-19T03:27:35.809 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-19T03:27:35.853 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T03:27:35.999 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package socat. 2020-11-19T03:27:36.014 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-19T03:27:36.057 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-19T03:27:36.132 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package xmlstarlet. 2020-11-19T03:27:36.147 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-19T03:27:36.193 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-19T03:27:36.436 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-19T03:27:36.451 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-19T03:27:36.494 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T03:27:36.552 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-test. 2020-11-19T03:27:36.568 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:36.611 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:36.780 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package xmlstarlet. 2020-11-19T03:27:36.795 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-19T03:27:36.838 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-19T03:27:37.170 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ceph-test. 2020-11-19T03:27:37.186 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:37.242 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:40.307 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-19T03:27:40.321 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:40.365 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:40.651 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package nvme-cli. 2020-11-19T03:27:40.667 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-19T03:27:40.709 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-19T03:27:40.989 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-19T03:27:41.006 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:41.041 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-attr. 2020-11-19T03:27:41.048 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:41.057 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-19T03:27:41.105 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-19T03:27:41.325 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package nvme-cli. 2020-11-19T03:27:41.342 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-19T03:27:41.374 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-19T03:27:41.408 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-funcsigs. 2020-11-19T03:27:41.422 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-19T03:27:41.474 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-19T03:27:41.705 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python-attr. 2020-11-19T03:27:41.721 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-19T03:27:41.762 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-19T03:27:41.785 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-pluggy. 2020-11-19T03:27:41.802 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-19T03:27:41.843 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-19T03:27:42.123 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python-funcsigs. 2020-11-19T03:27:42.137 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-19T03:27:42.154 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-py. 2020-11-19T03:27:42.171 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-19T03:27:42.182 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-19T03:27:42.213 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-19T03:27:42.468 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python-pluggy. 2020-11-19T03:27:42.483 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-19T03:27:42.526 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-19T03:27:42.549 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-pytest. 2020-11-19T03:27:42.566 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-19T03:27:42.607 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-19T03:27:42.837 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python-py. 2020-11-19T03:27:42.851 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-19T03:27:42.912 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-19T03:27:42.953 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-attr. 2020-11-19T03:27:42.970 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-19T03:27:43.011 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-19T03:27:43.274 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python-pytest. 2020-11-19T03:27:43.290 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-19T03:27:43.313 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cycler. 2020-11-19T03:27:43.329 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-19T03:27:43.332 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-19T03:27:43.372 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-19T03:27:43.686 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-attr. 2020-11-19T03:27:44.845 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webencodings. 2020-11-19T03:27:44.846 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-19T03:27:44.846 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-19T03:27:44.846 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-html5lib. 2020-11-19T03:27:44.847 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-19T03:27:44.847 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-19T03:27:44.847 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-19T03:27:44.848 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-19T03:27:44.848 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T03:27:44.849 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-19T03:27:44.849 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-19T03:27:44.849 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-cycler. 2020-11-19T03:27:44.851 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-19T03:27:44.851 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-19T03:27:44.851 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-webencodings. 2020-11-19T03:27:44.851 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-19T03:27:44.852 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-19T03:27:44.852 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-html5lib. 2020-11-19T03:27:44.852 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-19T03:27:44.853 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-19T03:27:45.001 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-19T03:27:45.014 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-19T03:27:45.059 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T03:27:45.215 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-19T03:27:45.229 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-19T03:27:45.290 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T03:27:45.386 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tz. 2020-11-19T03:27:45.401 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-19T03:27:45.444 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-19T03:27:45.718 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-19T03:27:45.732 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-19T03:27:45.764 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-19T03:27:45.777 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T03:27:45.781 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-19T03:27:45.822 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T03:27:46.113 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-tz. 2020-11-19T03:27:46.129 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-19T03:27:46.170 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-19T03:27:46.482 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-19T03:27:46.496 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-19T03:27:46.530 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-olefile. 2020-11-19T03:27:46.540 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T03:27:46.547 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-19T03:27:46.587 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-19T03:27:46.931 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastescript. 2020-11-19T03:27:46.948 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-19T03:27:46.990 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-19T03:27:47.238 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-olefile. 2020-11-19T03:27:47.253 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-19T03:27:47.297 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-19T03:27:47.360 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-19T03:27:47.377 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-19T03:27:47.419 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T03:27:47.650 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pastescript. 2020-11-19T03:27:47.664 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-19T03:27:47.708 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-19T03:27:47.846 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pluggy. 2020-11-19T03:27:47.861 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-19T03:27:47.912 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-19T03:27:48.036 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-19T03:27:48.050 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-19T03:27:48.094 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T03:27:48.215 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-py. 2020-11-19T03:27:48.230 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-19T03:27:48.274 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-19T03:27:48.489 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pluggy. 2020-11-19T03:27:48.506 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-19T03:27:48.548 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-19T03:27:48.577 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-19T03:27:48.594 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-19T03:27:48.877 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-py. 2020-11-19T03:27:48.877 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-19T03:27:48.877 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-19T03:27:48.918 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-19T03:27:48.963 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pytest. 2020-11-19T03:27:48.977 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-19T03:27:49.023 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-19T03:27:49.287 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-19T03:27:49.302 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-19T03:27:49.345 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-19T03:27:49.360 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplejson. 2020-11-19T03:27:49.374 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-19T03:27:49.419 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-19T03:27:49.656 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-pytest. 2020-11-19T03:27:49.673 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-19T03:27:49.715 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-19T03:27:49.771 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-19T03:27:49.785 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-19T03:27:49.837 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T03:27:50.084 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-simplejson. 2020-11-19T03:27:50.098 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-19T03:27:50.142 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-19T03:27:50.300 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package radosgw. 2020-11-19T03:27:50.316 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../110-radosgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:50.367 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:50.487 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-19T03:27:50.504 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-19T03:27:50.545 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T03:27:51.049 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package radosgw. 2020-11-19T03:27:51.064 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../110-radosgw_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:51.107 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:51.378 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package rbd-fuse. 2020-11-19T03:27:51.392 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:51.436 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:51.734 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-19T03:27:52.033 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T03:27:52.137 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package rbd-fuse. 2020-11-19T03:27:52.150 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7199-g409eac50-1bionic_amd64.deb ... 2020-11-19T03:27:52.194 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:52.409 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T03:27:52.512 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-19T03:27:52.535 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T03:27:52.840 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-19T03:27:53.154 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T03:27:53.208 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-19T03:27:53.334 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-19T03:27:53.549 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-19T03:27:53.832 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-19T03:27:54.450 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-19T03:27:54.450 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-19T03:27:54.451 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-19T03:27:54.451 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-19T03:27:54.462 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-19T03:27:54.631 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-19T03:27:54.815 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-19T03:27:54.941 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-19T03:27:55.142 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-19T03:27:55.167 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-19T03:27:55.268 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-19T03:27:55.481 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-19T03:27:55.619 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-19T03:27:55.782 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:55.961 INFO:teuthology.orchestra.run.smithi066.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-19T03:27:56.118 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-19T03:27:56.280 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-19T03:27:56.418 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-19T03:27:56.544 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T03:27:56.590 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-ceph-argparse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:27:56.908 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-19T03:27:57.250 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-19T03:27:57.376 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-19T03:27:57.489 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T03:27:57.823 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-19T03:27:58.158 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-py (1.5.2-1) ... 2020-11-19T03:27:58.338 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-19T03:27:58.385 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-19T03:27:58.664 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-19T03:27:58.711 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-19T03:27:59.323 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python-py (1.5.2-1) ... 2020-11-19T03:27:59.323 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-19T03:27:59.323 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-19T03:27:59.324 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-19T03:27:59.324 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-19T03:27:59.389 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-19T03:27:59.534 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-19T03:27:59.625 INFO:teuthology.orchestra.run.smithi066.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-19T03:27:59.690 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T03:27:59.878 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-19T03:28:00.004 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-19T03:28:00.008 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-19T03:28:00.230 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-19T03:28:00.293 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-19T03:28:00.539 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-19T03:28:00.652 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-19T03:28:00.865 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-19T03:28:01.004 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-19T03:28:01.200 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-19T03:28:01.321 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:01.584 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-19T03:28:01.919 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-19T03:28:01.997 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user cephadm....done 2020-11-19T03:28:02.161 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-19T03:28:02.220 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:02.446 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T03:28:02.572 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-19T03:28:02.714 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-19T03:28:02.914 INFO:teuthology.orchestra.run.smithi066.stdout:Adding system user cephadm....done 2020-11-19T03:28:03.035 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-19T03:28:03.066 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-19T03:28:03.302 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-19T03:28:03.359 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T03:28:03.420 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-19T03:28:03.488 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-19T03:28:03.546 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-19T03:28:03.820 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:04.241 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-19T03:28:04.241 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-19T03:28:04.242 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-19T03:28:04.333 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-19T03:28:04.414 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T03:28:04.556 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T03:28:04.676 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:04.899 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-19T03:28:04.977 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-19T03:28:05.261 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-19T03:28:05.267 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T03:28:05.357 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-19T03:28:05.387 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-19T03:28:05.389 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-19T03:28:05.628 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-19T03:28:05.738 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-19T03:28:05.937 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T03:28:06.115 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-19T03:28:06.204 INFO:teuthology.orchestra.run.smithi066.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-19T03:28:06.237 INFO:teuthology.orchestra.run.smithi066.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-19T03:28:06.238 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T03:28:06.475 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-19T03:28:06.589 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-19T03:28:06.760 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-19T03:28:06.907 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-19T03:28:07.046 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-19T03:28:07.126 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T03:28:07.403 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-19T03:28:07.543 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T03:28:07.633 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-19T03:28:07.704 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-19T03:28:07.895 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-19T03:28:07.931 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-19T03:28:08.071 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T03:28:08.231 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T03:28:08.349 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-19T03:28:08.439 INFO:teuthology.orchestra.run.smithi066.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-19T03:28:08.515 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T03:28:08.650 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-19T03:28:08.717 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-19T03:28:08.897 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-19T03:28:08.978 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-19T03:28:09.035 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T03:28:09.177 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-19T03:28:09.204 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-19T03:28:09.330 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-19T03:28:09.453 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-19T03:28:09.731 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-19T03:28:09.738 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T03:28:09.857 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-19T03:28:10.014 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-19T03:28:10.158 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-19T03:28:10.315 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-19T03:28:10.493 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-19T03:28:10.683 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-19T03:28:10.769 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-19T03:28:10.992 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-19T03:28:11.062 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-19T03:28:11.302 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-19T03:28:11.430 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-19T03:28:11.620 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T03:28:11.756 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-19T03:28:11.963 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-19T03:28:12.087 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-19T03:28:12.272 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-19T03:28:12.398 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T03:28:12.416 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-19T03:28:12.787 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-19T03:28:13.069 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-19T03:28:13.170 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-19T03:28:13.562 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T03:28:14.123 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-19T03:28:14.442 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-19T03:28:14.690 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-19T03:28:14.752 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-19T03:28:14.825 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-19T03:28:15.053 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-19T03:28:15.154 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-19T03:28:15.192 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-19T03:28:15.468 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T03:28:15.522 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-19T03:28:15.578 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-19T03:28:15.695 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librados2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:15.782 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-19T03:28:15.796 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-19T03:28:15.908 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-19T03:28:16.033 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up librados2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:16.089 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:16.159 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-19T03:28:16.215 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:16.374 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:16.444 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libcephfs2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:16.500 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T03:28:16.570 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libcephfs-dev (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:16.673 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-rados (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:16.797 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-19T03:28:16.820 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-19T03:28:17.115 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-19T03:28:17.127 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-19T03:28:17.416 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-19T03:28:17.420 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T03:28:17.753 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-19T03:28:18.375 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-19T03:28:18.698 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-19T03:28:19.188 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T03:28:20.346 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-19T03:28:20.347 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.347 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.347 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.347 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.348 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libradosstriper1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.348 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cephfs (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.348 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librgw2 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.348 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.349 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T03:28:20.349 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T03:28:20.560 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T03:28:20.561 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-19T03:28:20.593 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librbd1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.703 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-19T03:28:20.776 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-19T03:28:20.808 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-19T03:28:20.856 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up librbd1 (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:20.982 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-19T03:28:21.072 INFO:teuthology.orchestra.run.smithi066.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-19T03:28:21.098 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T03:28:21.112 INFO:teuthology.orchestra.run.smithi066.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-19T03:28:21.427 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-19T03:28:21.775 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:21.901 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-19T03:28:22.088 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mgr-modules-core (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.213 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-19T03:28:22.219 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.320 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.421 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.540 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-rgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.555 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.666 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up rbd-fuse (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.809 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up python3-rbd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:22.935 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-common (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:23.089 INFO:teuthology.orchestra.run.smithi154.stdout:Adding group ceph....done 2020-11-19T03:28:23.402 INFO:teuthology.orchestra.run.smithi066.stdout:Adding group ceph....done 2020-11-19T03:28:23.442 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user ceph....done 2020-11-19T03:28:23.654 INFO:teuthology.orchestra.run.smithi154.stdout:Setting system user ceph properties....done 2020-11-19T03:28:23.816 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-19T03:28:24.870 INFO:teuthology.orchestra.run.smithi066.stdout:Adding system user ceph....done 2020-11-19T03:28:24.870 INFO:teuthology.orchestra.run.smithi066.stdout:Setting system user ceph properties....done 2020-11-19T03:28:24.871 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-19T03:28:24.871 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-19T03:28:24.872 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:24.873 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-19T03:28:24.873 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up radosgw (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:24.876 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T03:28:24.877 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T03:28:24.933 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T03:28:24.934 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-19T03:28:25.151 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:25.277 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:25.321 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-test (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:25.456 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-base (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:25.591 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-19T03:28:25.792 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-19T03:28:25.879 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:26.078 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T03:28:26.078 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T03:28:26.082 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mgr (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:26.305 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T03:28:26.305 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-19T03:28:26.363 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:26.506 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:26.617 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:26.785 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-osd (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:26.997 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T03:28:26.997 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T03:28:27.225 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T03:28:27.225 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-19T03:28:27.267 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:27.410 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mgr-cephadm (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:27.552 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:27.622 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mgr-rook (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:27.773 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mds (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:27.792 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T03:28:27.793 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T03:28:27.997 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T03:28:27.998 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-19T03:28:28.062 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:28.263 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T03:28:28.264 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T03:28:28.282 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mon (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:28.506 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T03:28:28.507 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-19T03:28:28.521 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:28.658 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:28.784 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:29.239 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:29.239 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-19T03:28:29.240 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-19T03:28:29.240 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph-mgr-dashboard (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:29.240 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ceph (16.0.0-7199-g409eac50-1bionic) ... 2020-11-19T03:28:29.240 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-19T03:28:29.447 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-19T03:28:31.196 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-19T03:28:31.231 INFO:teuthology.orchestra.run.smithi154.stdout:ureadahead will be reprofiled on next reboot 2020-11-19T03:28:31.279 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-19T03:28:31.516 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-19T03:28:31.559 INFO:teuthology.orchestra.run.smithi066.stdout:ureadahead will be reprofiled on next reboot 2020-11-19T03:28:31.608 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-19T03:28:31.721 INFO:teuthology.orchestra.run.smithi154.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-19T03:28:31.723 DEBUG:teuthology.parallel:result is None 2020-11-19T03:28:32.052 INFO:teuthology.orchestra.run.smithi066.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-19T03:28:32.055 DEBUG:teuthology.parallel:result is None 2020-11-19T03:28:32.056 INFO:teuthology.orchestra.run.smithi066:> sudo lsb_release -is 2020-11-19T03:28:32.107 INFO:teuthology.orchestra.run.smithi066.stdout:Ubuntu 2020-11-19T03:28:32.112 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T03:28:32.112 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T03:28:32.284 INFO:teuthology.orchestra.run.smithi066:> dpkg-query -W -f '${Version}' ceph 2020-11-19T03:28:32.308 INFO:teuthology.orchestra.run.smithi066.stdout:16.0.0-7199-g409eac50-1bionic 2020-11-19T03:28:32.309 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7199-g409eac50-1bionic 2020-11-19T03:28:32.309 INFO:teuthology.task.install:The correct ceph version 16.0.0-7199-g409eac50-1bionic is installed. 2020-11-19T03:28:32.310 INFO:teuthology.orchestra.run.smithi154:> sudo lsb_release -is 2020-11-19T03:28:32.365 INFO:teuthology.orchestra.run.smithi154.stdout:Ubuntu 2020-11-19T03:28:32.370 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-19T03:28:32.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=409eac507e130adcc9fee71aa5fd16ed7f289313 2020-11-19T03:28:32.536 INFO:teuthology.orchestra.run.smithi154:> dpkg-query -W -f '${Version}' ceph 2020-11-19T03:28:32.560 INFO:teuthology.orchestra.run.smithi154.stdout:16.0.0-7199-g409eac50-1bionic 2020-11-19T03:28:32.561 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7199-g409eac50-1bionic 2020-11-19T03:28:32.561 INFO:teuthology.task.install:The correct ceph version 16.0.0-7199-g409eac50-1bionic is installed. 2020-11-19T03:28:32.562 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-19T03:28:32.562 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:32.563 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T03:28:32.568 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T03:28:32.577 INFO:teuthology.orchestra.run.smithi066.stderr:24+1 records in 2020-11-19T03:28:32.577 INFO:teuthology.orchestra.run.smithi066.stderr:24+1 records out 2020-11-19T03:28:32.578 INFO:teuthology.orchestra.run.smithi066.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000216341 s, 58.6 MB/s 2020-11-19T03:28:32.578 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:28:32.579 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T03:28:32.617 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-19T03:28:32.620 INFO:teuthology.orchestra.run.smithi154.stderr:24+1 records in 2020-11-19T03:28:32.621 INFO:teuthology.orchestra.run.smithi154.stderr:24+1 records out 2020-11-19T03:28:32.621 INFO:teuthology.orchestra.run.smithi154.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000372087 s, 34.0 MB/s 2020-11-19T03:28:32.623 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-19T03:28:32.624 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:32.624 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/daemon-helper 2020-11-19T03:28:32.635 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-19T03:28:32.641 INFO:teuthology.orchestra.run.smithi066.stderr:5+1 records in 2020-11-19T03:28:32.642 INFO:teuthology.orchestra.run.smithi066.stderr:5+1 records out 2020-11-19T03:28:32.642 INFO:teuthology.orchestra.run.smithi066.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.000101267 s, 30.2 MB/s 2020-11-19T03:28:32.644 INFO:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-19T03:28:32.702 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:28:32.702 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2020-11-19T03:28:32.720 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-19T03:28:32.729 INFO:teuthology.orchestra.run.smithi154.stderr:5+1 records in 2020-11-19T03:28:32.729 INFO:teuthology.orchestra.run.smithi154.stderr:5+1 records out 2020-11-19T03:28:32.730 INFO:teuthology.orchestra.run.smithi154.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.956e-05 s, 30.7 MB/s 2020-11-19T03:28:32.731 INFO:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-19T03:28:32.792 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-19T03:28:32.792 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:32.793 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T03:28:32.801 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T03:28:32.811 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:32.812 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:32.812 INFO:teuthology.orchestra.run.smithi066.stderr:299 bytes copied, 7.0827e-05 s, 4.2 MB/s 2020-11-19T03:28:32.813 INFO:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-19T03:28:32.871 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:28:32.872 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T03:28:32.880 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-19T03:28:32.888 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:28:32.889 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:28:32.889 INFO:teuthology.orchestra.run.smithi154.stderr:299 bytes copied, 6.97e-05 s, 4.3 MB/s 2020-11-19T03:28:32.891 INFO:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-19T03:28:32.949 INFO:teuthology.run_tasks:Running task ceph... 2020-11-19T03:28:33.011 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-19T03:28:33.012 INFO:teuthology.orchestra.run.smithi066:> sudo chmod 777 /var/log/ceph 2020-11-19T03:28:33.017 INFO:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2020-11-19T03:28:33.034 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-19T03:28:33.034 INFO:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-19T03:28:33.076 INFO:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-19T03:28:33.093 INFO:tasks.ceph:Creating extra log directories... 2020-11-19T03:28:33.093 INFO:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-19T03:28:33.132 INFO:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-19T03:28:33.255 INFO:tasks.ceph:Creating ceph cluster c1... 2020-11-19T03:28:33.255 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-19T03:28:33.255 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633886', 'branch': 'wip-yuval-fix-47904', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5633886', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuval-fix-47904'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuval-fix-47904', 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuval-fix-47904', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.1594', 'targets': {'smithi066.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcRZ1FksMeIk6CFCb3W6w2tfikSMp9EDkwlSGqUYBi2RLjZJn2SE1rITizpE4X5hGFzRU8SdaWIy3iuhD4VIPrEIa7/SFpk9OfJzkorD6I72swHFJKBahLLH6aJX5Yx8/I7BLiYXzJ0+xyEfobvaZZVf/7gyHldHTDYOU9Lls45PF0S/RN/ZXE+S16ME6+Ef0pIfkwBENcFDcKBdFcL5pk0z6jedB7GMwwrXsGd6EBcOUYxhR6y+wxkMgHMSBVyflVwTK85snOI6shbg9PrvqdnZRkv24i1Kc2lE33YwhXgnp1lLhSceER2/FXlgmz2th2cIsnFOCDY7u12H8T+knj', 'smithi154.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5EGCCacrU4DTfSf3En71+Jcujq6iYhrNBfANMGFlLcrKHyhaRuoNs89UKHMHPSgyJL+hi/5JStDf/DpnvV3aXcLCOL414b4ohNF0Gfrc6PGDmt3WNAjLNr3pweEUgkYpyRFH4f5snFluElHQJ+KdwuU6GAS/c/a87rPkLUAyCa0LaxR5x3dMC+MXH7nRmaOITXjAQNO8e5u1AZPubTFb73nWWHmeBD3tSav1CVI8x1zYgVIfIDrHTeLQwy4dviozplDeUZP/GL0yvq2RxTEbUdWhytvVgnEyCCcuyXmAQaaCc+K8k0Rsw3UqUnhNr1XV/dqaxHFKeW61wcgo1B8GR'}} 2020-11-19T03:28:33.256 INFO:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-19T03:28:33.259 INFO:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-19T03:28:33.266 INFO:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T03:28:33.307 INFO:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T03:28:33.324 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:33.324 INFO:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2020-11-19T03:28:33.366 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-19T03:28:33.367 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_1 2020-11-19T03:28:33.367 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_2 2020-11-19T03:28:33.368 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_3 2020-11-19T03:28:33.368 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_4 2020-11-19T03:28:33.368 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:33.368 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:33.369 INFO:teuthology.orchestra.run.smithi066.stderr:72 bytes copied, 0.00018001 s, 400 kB/s 2020-11-19T03:28:33.369 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-19T03:28:33.369 INFO:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-19T03:28:33.416 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-19T03:28:33.416 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:28:33.416 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 6h/6d Inode: 343 Links: 1 2020-11-19T03:28:33.417 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:28:33.417 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2020-11-19 03:26:06.605135873 +0000 2020-11-19T03:28:33.417 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2020-11-19 03:26:06.601135873 +0000 2020-11-19T03:28:33.417 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2020-11-19 03:26:06.601135873 +0000 2020-11-19T03:28:33.417 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2020-11-19T03:28:33.426 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2020-11-19T03:28:33.426 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2020-11-19T03:28:33.426 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000362665 s, 1.4 MB/s 2020-11-19T03:28:33.430 INFO:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-19T03:28:33.487 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-19T03:28:33.487 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:28:33.487 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 6h/6d Inode: 350 Links: 1 2020-11-19T03:28:33.488 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:28:33.488 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2020-11-19 03:26:07.001135868 +0000 2020-11-19T03:28:33.488 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2020-11-19 03:26:06.893135870 +0000 2020-11-19T03:28:33.488 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2020-11-19 03:26:06.893135870 +0000 2020-11-19T03:28:33.488 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2020-11-19T03:28:33.496 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2020-11-19T03:28:33.497 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2020-11-19T03:28:33.497 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000361227 s, 1.4 MB/s 2020-11-19T03:28:33.502 INFO:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-19T03:28:33.560 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-19T03:28:33.560 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:28:33.561 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 6h/6d Inode: 356 Links: 1 2020-11-19T03:28:33.561 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:28:33.561 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2020-11-19 03:26:07.253135866 +0000 2020-11-19T03:28:33.562 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2020-11-19 03:26:07.149135867 +0000 2020-11-19T03:28:33.562 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2020-11-19 03:26:07.149135867 +0000 2020-11-19T03:28:33.563 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2020-11-19T03:28:33.570 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2020-11-19T03:28:33.571 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2020-11-19T03:28:33.571 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000302695 s, 1.7 MB/s 2020-11-19T03:28:33.574 INFO:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-19T03:28:33.595 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-19T03:28:33.595 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:28:33.596 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 6h/6d Inode: 362 Links: 1 2020-11-19T03:28:33.596 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:28:33.596 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2020-11-19 03:26:07.481135863 +0000 2020-11-19T03:28:33.596 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2020-11-19 03:26:07.389135864 +0000 2020-11-19T03:28:33.597 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2020-11-19 03:26:07.389135864 +0000 2020-11-19T03:28:33.597 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2020-11-19T03:28:33.611 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2020-11-19T03:28:33.612 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2020-11-19T03:28:33.614 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000382794 s, 1.3 MB/s 2020-11-19T03:28:33.616 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:28:33.617 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T03:28:33.617 INFO:tasks.ceph:Generating config... 2020-11-19T03:28:33.620 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-19T03:28:33.620 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-19T03:28:33.621 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-19T03:28:33.621 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T03:28:33.622 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-19T03:28:33.622 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-19T03:28:33.623 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-19T03:28:33.623 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-19T03:28:33.624 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-19T03:28:33.624 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-19T03:28:33.625 INFO:tasks.ceph:[client] setuser = ceph 2020-11-19T03:28:33.626 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-19T03:28:33.626 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-19T03:28:33.627 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-19T03:28:33.627 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-19T03:28:33.628 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-19T03:28:33.628 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-19T03:28:33.629 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-19T03:28:33.629 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-19T03:28:33.630 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-19T03:28:33.630 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-19T03:28:33.631 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-19T03:28:33.631 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-19T03:28:33.632 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-19T03:28:33.633 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-19T03:28:33.633 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-19T03:28:33.634 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-19T03:28:33.634 INFO:tasks.ceph:Setting up c1.mon.a... 2020-11-19T03:28:33.635 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-11-19T03:28:33.697 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/c1.keyring 2020-11-19T03:28:34.987 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-11-19T03:28:35.064 INFO:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T03:28:35.120 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.66')] 2020-11-19T03:28:35.120 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.66', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-19T03:28:35.121 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:35.122 INFO:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T03:28:35.170 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T03:28:35.171 INFO:teuthology.orchestra.run.smithi066.stderr:6+1 records in 2020-11-19T03:28:35.172 INFO:teuthology.orchestra.run.smithi066.stderr:6+1 records out 2020-11-19T03:28:35.172 INFO:teuthology.orchestra.run.smithi066.stderr:3135 bytes (3.1 kB, 3.1 KiB) copied, 0.000108467 s, 28.9 MB/s 2020-11-19T03:28:35.172 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.66 --print /home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:35.244 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:28:35.245 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:28:35.245 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:35.245 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: generated fsid f9ed144c-5db3-4d83-993d-51016a1aaef3 2020-11-19T03:28:35.245 INFO:teuthology.orchestra.run.smithi066.stdout:epoch 0 2020-11-19T03:28:35.246 INFO:teuthology.orchestra.run.smithi066.stdout:fsid f9ed144c-5db3-4d83-993d-51016a1aaef3 2020-11-19T03:28:35.246 INFO:teuthology.orchestra.run.smithi066.stdout:last_changed 2020-11-19T03:28:35.240520+0000 2020-11-19T03:28:35.246 INFO:teuthology.orchestra.run.smithi066.stdout:created 2020-11-19T03:28:35.240520+0000 2020-11-19T03:28:35.246 INFO:teuthology.orchestra.run.smithi066.stdout:min_mon_release 0 (unknown) 2020-11-19T03:28:35.247 INFO:teuthology.orchestra.run.smithi066.stdout:election_strategy: 1 2020-11-19T03:28:35.247 INFO:teuthology.orchestra.run.smithi066.stdout:0: [v2:172.21.15.66:3300/0,v1:172.21.15.66:6789/0] mon.a 2020-11-19T03:28:35.247 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-11-19T03:28:35.248 INFO:teuthology.orchestra.run.smithi066:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T03:28:35.295 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID f9ed144c-5db3-4d83-993d-51016a1aaef3... 2020-11-19T03:28:35.297 INFO:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-19T03:28:35.339 INFO:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-19T03:28:35.369 INFO:teuthology.orchestra.run.smithi066.stdout:[global] 2020-11-19T03:28:35.369 INFO:teuthology.orchestra.run.smithi066.stdout: chdir = "" 2020-11-19T03:28:35.369 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T03:28:35.370 INFO:teuthology.orchestra.run.smithi066.stdout: auth supported = cephx 2020-11-19T03:28:35.370 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.370 INFO:teuthology.orchestra.run.smithi066.stdout: filestore xattr use omap = true 2020-11-19T03:28:35.370 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.371 INFO:teuthology.orchestra.run.smithi066.stdout: mon clock drift allowed = 1.000 2020-11-19T03:28:35.371 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.371 INFO:teuthology.orchestra.run.smithi066.stdout: osd crush chooseleaf type = 0 2020-11-19T03:28:35.371 INFO:teuthology.orchestra.run.smithi066.stdout: auth debug = true 2020-11-19T03:28:35.371 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.372 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on old message = true 2020-11-19T03:28:35.372 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on bug = true 2020-11-19T03:28:35.372 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.372 INFO:teuthology.orchestra.run.smithi066.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T03:28:35.372 INFO:teuthology.orchestra.run.smithi066.stdout: mon pg warn max object skew = 0 2020-11-19T03:28:35.373 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.373 INFO:teuthology.orchestra.run.smithi066.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T03:28:35.373 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T03:28:35.373 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.373 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default size = 2 2020-11-19T03:28:35.374 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.374 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow primary affinity = true 2020-11-19T03:28:35.374 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow pg remap = true 2020-11-19T03:28:35.374 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on legacy crush tunables = false 2020-11-19T03:28:35.374 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on crush straw calc version zero = false 2020-11-19T03:28:35.375 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on no sortbitwise = false 2020-11-19T03:28:35.375 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on osd down out interval zero = false 2020-11-19T03:28:35.375 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on too few osds = false 2020-11-19T03:28:35.375 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T03:28:35.375 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T03:28:35.376 INFO:teuthology.orchestra.run.smithi066.stdout: mon_allow_pool_size_one = true 2020-11-19T03:28:35.376 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.376 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T03:28:35.376 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.376 INFO:teuthology.orchestra.run.smithi066.stdout: osd default data pool replay window = 5 2020-11-19T03:28:35.377 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.377 INFO:teuthology.orchestra.run.smithi066.stdout: mon allow pool delete = true 2020-11-19T03:28:35.377 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.377 INFO:teuthology.orchestra.run.smithi066.stdout: mon cluster log file level = debug 2020-11-19T03:28:35.377 INFO:teuthology.orchestra.run.smithi066.stdout: debug asserts on shutdown = true 2020-11-19T03:28:35.378 INFO:teuthology.orchestra.run.smithi066.stdout: mon health detail to clog = false 2020-11-19T03:28:35.378 INFO:teuthology.orchestra.run.smithi066.stdout: mon host = 172.21.15.66 2020-11-19T03:28:35.378 INFO:teuthology.orchestra.run.smithi066.stdout: osd heartbeat grace = 40 2020-11-19T03:28:35.378 INFO:teuthology.orchestra.run.smithi066.stdout: fsid = f9ed144c-5db3-4d83-993d-51016a1aaef3 2020-11-19T03:28:35.378 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.379 INFO:teuthology.orchestra.run.smithi066.stdout:[osd] 2020-11-19T03:28:35.379 INFO:teuthology.orchestra.run.smithi066.stdout: osd journal size = 100 2020-11-19T03:28:35.379 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.379 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub load threshold = 5.0 2020-11-19T03:28:35.379 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub max interval = 600 2020-11-19T03:28:35.379 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.380 INFO:teuthology.orchestra.run.smithi066.stdout: osd recover clone overlap = true 2020-11-19T03:28:35.380 INFO:teuthology.orchestra.run.smithi066.stdout: osd recovery max chunk = 1048576 2020-11-19T03:28:35.380 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.380 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug shutdown = true 2020-11-19T03:28:35.381 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug op order = true 2020-11-19T03:28:35.381 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify stray on activate = true 2020-11-19T03:28:35.381 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.381 INFO:teuthology.orchestra.run.smithi066.stdout: osd open classes on start = true 2020-11-19T03:28:35.381 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug pg log writeout = true 2020-11-19T03:28:35.382 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.382 INFO:teuthology.orchestra.run.smithi066.stdout: osd deep scrub update digest min age = 30 2020-11-19T03:28:35.382 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.382 INFO:teuthology.orchestra.run.smithi066.stdout: osd map max advance = 10 2020-11-19T03:28:35.382 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.382 INFO:teuthology.orchestra.run.smithi066.stdout: journal zero on create = true 2020-11-19T03:28:35.383 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.383 INFO:teuthology.orchestra.run.smithi066.stdout: filestore ondisk finisher threads = 3 2020-11-19T03:28:35.383 INFO:teuthology.orchestra.run.smithi066.stdout: filestore apply finisher threads = 3 2020-11-19T03:28:35.383 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.384 INFO:teuthology.orchestra.run.smithi066.stdout: bdev debug aio = true 2020-11-19T03:28:35.384 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug misdirected ops = true 2020-11-19T03:28:35.384 INFO:teuthology.orchestra.run.smithi066.stdout: debug filestore = 20 2020-11-19T03:28:35.384 INFO:teuthology.orchestra.run.smithi066.stdout: debug journal = 20 2020-11-19T03:28:35.384 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 20 2020-11-19T03:28:35.385 INFO:teuthology.orchestra.run.smithi066.stdout: debug osd = 25 2020-11-19T03:28:35.385 INFO:teuthology.orchestra.run.smithi066.stdout: osd fast shutdown = False 2020-11-19T03:28:35.385 INFO:teuthology.orchestra.run.smithi066.stdout: rocksdb delete range threshold = 0 2020-11-19T03:28:35.385 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.385 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr] 2020-11-19T03:28:35.386 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2020-11-19T03:28:35.386 INFO:teuthology.orchestra.run.smithi066.stdout: debug mgr = 20 2020-11-19T03:28:35.386 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2020-11-19T03:28:35.386 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2020-11-19T03:28:35.386 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:28:35.387 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:28:35.387 INFO:teuthology.orchestra.run.smithi066.stdout: mgr/telemetry/nag = false 2020-11-19T03:28:35.387 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.387 INFO:teuthology.orchestra.run.smithi066.stdout:[mon] 2020-11-19T03:28:35.387 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2020-11-19T03:28:35.388 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2020-11-19T03:28:35.388 INFO:teuthology.orchestra.run.smithi066.stdout: debug paxos = 20 2020-11-19T03:28:35.388 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2020-11-19T03:28:35.388 INFO:teuthology.orchestra.run.smithi066.stdout: mon data avail warn = 5 2020-11-19T03:28:35.388 INFO:teuthology.orchestra.run.smithi066.stdout: mon mgr mkfs grace = 240 2020-11-19T03:28:35.389 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:28:35.389 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd reporter subtree level = osd 2020-11-19T03:28:35.389 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd prime pg temp = true 2020-11-19T03:28:35.389 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:28:35.389 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd crush smoke test = False 2020-11-19T03:28:35.390 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:35.390 INFO:teuthology.orchestra.run.smithi066.stdout:[client] 2020-11-19T03:28:35.390 INFO:teuthology.orchestra.run.smithi066.stdout: rgw cache enabled = true 2020-11-19T03:28:35.390 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable ops log = true 2020-11-19T03:28:35.390 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable usage log = true 2020-11-19T03:28:35.391 INFO:teuthology.orchestra.run.smithi066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T03:28:35.391 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T03:28:35.391 INFO:teuthology.orchestra.run.smithi066.stdout: debug rgw = 20 2020-11-19T03:28:35.391 INFO:teuthology.orchestra.run.smithi066.stdout: rgw crypt require ssl = False 2020-11-19T03:28:35.391 INFO:teuthology.orchestra.run.smithi066.stdout: rgw crypt s3 kms backend = testing 2020-11-19T03:28:35.392 INFO:teuthology.orchestra.run.smithi066.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T03:28:35.392 INFO:teuthology.orchestra.run.smithi066.stdout: rgw curl low speed time = 300 2020-11-19T03:28:35.392 INFO:teuthology.orchestra.run.smithi066.stdout: rgw data log num shards = 4 2020-11-19T03:28:35.392 INFO:teuthology.orchestra.run.smithi066.stdout: rgw md log max shards = 4 2020-11-19T03:28:35.392 INFO:teuthology.orchestra.run.smithi066.stdout: rgw sync log trim interval = 0 2020-11-19T03:28:35.393 INFO:teuthology.orchestra.run.smithi066.stdout: rgw sync obj etag verify = True 2020-11-19T03:28:35.393 INFO:teuthology.orchestra.run.smithi066.stdout: setgroup = ceph 2020-11-19T03:28:35.393 INFO:teuthology.orchestra.run.smithi066.stdout: setuser = ceph 2020-11-19T03:28:35.393 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.a] 2020-11-19T03:28:35.393 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.0] 2020-11-19T03:28:35.394 INFO:teuthology.orchestra.run.smithi066.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T03:28:35.394 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.1] 2020-11-19T03:28:35.394 INFO:teuthology.orchestra.run.smithi066.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T03:28:35.394 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.2] 2020-11-19T03:28:35.394 INFO:teuthology.orchestra.run.smithi066.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T03:28:35.396 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2020-11-19T03:28:35.396 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2020-11-19T03:28:35.396 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T03:28:35.397 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2020-11-19T03:28:35.397 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.397 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2020-11-19T03:28:35.397 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.397 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2020-11-19T03:28:35.398 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.398 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2020-11-19T03:28:35.398 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2020-11-19T03:28:35.398 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.398 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2020-11-19T03:28:35.399 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2020-11-19T03:28:35.399 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.399 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T03:28:35.399 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2020-11-19T03:28:35.399 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.400 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T03:28:35.400 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T03:28:35.400 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.400 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2020-11-19T03:28:35.401 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.401 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2020-11-19T03:28:35.402 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2020-11-19T03:28:35.402 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2020-11-19T03:28:35.402 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2020-11-19T03:28:35.403 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2020-11-19T03:28:35.403 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2020-11-19T03:28:35.403 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2020-11-19T03:28:35.404 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T03:28:35.404 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T03:28:35.405 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2020-11-19T03:28:35.405 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.405 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T03:28:35.406 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.406 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2020-11-19T03:28:35.407 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.407 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2020-11-19T03:28:35.407 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.407 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2020-11-19T03:28:35.407 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2020-11-19T03:28:35.408 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2020-11-19T03:28:35.408 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = 172.21.15.66 2020-11-19T03:28:35.408 INFO:teuthology.orchestra.run.smithi154.stdout: osd heartbeat grace = 40 2020-11-19T03:28:35.408 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = f9ed144c-5db3-4d83-993d-51016a1aaef3 2020-11-19T03:28:35.408 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.409 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2020-11-19T03:28:35.409 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2020-11-19T03:28:35.409 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.409 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2020-11-19T03:28:35.409 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2020-11-19T03:28:35.410 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.410 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2020-11-19T03:28:35.410 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2020-11-19T03:28:35.410 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.410 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2020-11-19T03:28:35.411 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2020-11-19T03:28:35.411 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2020-11-19T03:28:35.411 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.411 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2020-11-19T03:28:35.411 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2020-11-19T03:28:35.412 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.412 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2020-11-19T03:28:35.412 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.412 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2020-11-19T03:28:35.412 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.413 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2020-11-19T03:28:35.413 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.413 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2020-11-19T03:28:35.413 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2020-11-19T03:28:35.413 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.414 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2020-11-19T03:28:35.414 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2020-11-19T03:28:35.414 INFO:teuthology.orchestra.run.smithi154.stdout: debug filestore = 20 2020-11-19T03:28:35.414 INFO:teuthology.orchestra.run.smithi154.stdout: debug journal = 20 2020-11-19T03:28:35.414 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 20 2020-11-19T03:28:35.415 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 25 2020-11-19T03:28:35.415 INFO:teuthology.orchestra.run.smithi154.stdout: osd fast shutdown = False 2020-11-19T03:28:35.415 INFO:teuthology.orchestra.run.smithi154.stdout: rocksdb delete range threshold = 0 2020-11-19T03:28:35.415 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.415 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2020-11-19T03:28:35.416 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2020-11-19T03:28:35.416 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2020-11-19T03:28:35.416 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2020-11-19T03:28:35.416 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2020-11-19T03:28:35.416 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:28:35.417 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:28:35.417 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2020-11-19T03:28:35.417 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.417 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2020-11-19T03:28:35.417 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2020-11-19T03:28:35.418 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2020-11-19T03:28:35.418 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2020-11-19T03:28:35.418 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2020-11-19T03:28:35.418 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2020-11-19T03:28:35.418 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2020-11-19T03:28:35.419 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:28:35.419 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2020-11-19T03:28:35.419 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2020-11-19T03:28:35.419 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:28:35.419 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd crush smoke test = False 2020-11-19T03:28:35.420 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:28:35.420 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2020-11-19T03:28:35.420 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2020-11-19T03:28:35.420 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2020-11-19T03:28:35.420 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2020-11-19T03:28:35.421 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T03:28:35.421 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T03:28:35.421 INFO:teuthology.orchestra.run.smithi154.stdout: debug rgw = 20 2020-11-19T03:28:35.421 INFO:teuthology.orchestra.run.smithi154.stdout: rgw crypt require ssl = False 2020-11-19T03:28:35.421 INFO:teuthology.orchestra.run.smithi154.stdout: rgw crypt s3 kms backend = testing 2020-11-19T03:28:35.422 INFO:teuthology.orchestra.run.smithi154.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T03:28:35.422 INFO:teuthology.orchestra.run.smithi154.stdout: rgw curl low speed time = 300 2020-11-19T03:28:35.422 INFO:teuthology.orchestra.run.smithi154.stdout: rgw data log num shards = 4 2020-11-19T03:28:35.422 INFO:teuthology.orchestra.run.smithi154.stdout: rgw md log max shards = 4 2020-11-19T03:28:35.422 INFO:teuthology.orchestra.run.smithi154.stdout: rgw sync log trim interval = 0 2020-11-19T03:28:35.423 INFO:teuthology.orchestra.run.smithi154.stdout: rgw sync obj etag verify = True 2020-11-19T03:28:35.423 INFO:teuthology.orchestra.run.smithi154.stdout: setgroup = ceph 2020-11-19T03:28:35.423 INFO:teuthology.orchestra.run.smithi154.stdout: setuser = ceph 2020-11-19T03:28:35.423 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2020-11-19T03:28:35.423 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2020-11-19T03:28:35.424 INFO:teuthology.orchestra.run.smithi154.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T03:28:35.424 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2020-11-19T03:28:35.424 INFO:teuthology.orchestra.run.smithi154.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T03:28:35.424 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2020-11-19T03:28:35.424 INFO:teuthology.orchestra.run.smithi154.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T03:28:35.425 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-11-19T03:28:35.425 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-11-19T03:28:35.484 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-19T03:28:35.485 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:35.486 INFO:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-19T03:28:35.490 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-19T03:28:35.491 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.] 2020-11-19T03:28:35.491 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVf/yBnAxAAxbe72RCcqqu1TsYX0Wqvng== 2020-11-19T03:28:35.492 INFO:teuthology.orchestra.run.smithi066.stdout:[client.admin] 2020-11-19T03:28:35.492 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVfPcp2HBAAmao17fExYNI6XAkGParSFw== 2020-11-19T03:28:35.492 INFO:teuthology.orchestra.run.smithi066.stdout: caps mds = "allow *" 2020-11-19T03:28:35.492 INFO:teuthology.orchestra.run.smithi066.stdout: caps mgr = "allow *" 2020-11-19T03:28:35.493 INFO:teuthology.orchestra.run.smithi066.stdout: caps mon = "allow *" 2020-11-19T03:28:35.493 INFO:teuthology.orchestra.run.smithi066.stdout: caps osd = "allow *" 2020-11-19T03:28:35.493 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:35.493 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:35.494 INFO:teuthology.orchestra.run.smithi066.stderr:206 bytes copied, 9.4233e-05 s, 2.2 MB/s 2020-11-19T03:28:35.494 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:35.494 INFO:teuthology.orchestra.run.smithi066:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-19T03:28:35.538 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-19T03:28:35.539 INFO:teuthology.orchestra.run.smithi066.stdout: ���L]�M��=Qj��c�_J Vc�_J V�aXa �B��Ba 2020-11-19T03:28:35.539 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:35.540 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:35.540 INFO:teuthology.orchestra.run.smithi066.stderr:201 bytes copied, 0.000176393 s, 1.1 MB/s 2020-11-19T03:28:35.540 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:28:35.540 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:35.541 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/c1.keyring 2020-11-19T03:28:35.541 INFO:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T03:28:35.586 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-19T03:28:35.596 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:35.597 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:35.597 INFO:teuthology.orchestra.run.smithi066.stderr:206 bytes copied, 0.000133917 s, 1.5 MB/s 2020-11-19T03:28:35.598 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T03:28:35.607 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:35.608 INFO:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:35.654 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:35.655 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:35.656 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:35.656 INFO:teuthology.orchestra.run.smithi066.stderr:201 bytes copied, 0.00013451 s, 1.5 MB/s 2020-11-19T03:28:35.656 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:28:35.657 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:28:35.657 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/c1.keyring 2020-11-19T03:28:35.658 INFO:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T03:28:35.663 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-19T03:28:35.672 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:28:35.672 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:28:35.672 INFO:teuthology.orchestra.run.smithi154.stderr:206 bytes copied, 6.8571e-05 s, 3.0 MB/s 2020-11-19T03:28:35.673 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-19T03:28:35.682 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:28:35.683 INFO:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:35.730 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:35.732 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:28:35.732 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:28:35.732 INFO:teuthology.orchestra.run.smithi154.stderr:201 bytes copied, 6.4166e-05 s, 3.1 MB/s 2020-11-19T03:28:35.733 INFO:tasks.ceph:Setting up mon nodes... 2020-11-19T03:28:35.733 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-19T03:28:35.734 INFO:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-11-19T03:28:35.775 INFO:teuthology.orchestra.run.smithi066.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-11-19T03:28:35.777 INFO:tasks.ceph:Setting up mds nodes... 2020-11-19T03:28:35.778 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-19T03:28:35.778 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-11-19T03:28:35.850 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/c1.client.0.keyring 2020-11-19T03:28:35.862 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-11-19T03:28:35.938 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/c1.client.1.keyring 2020-11-19T03:28:35.951 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-19T03:28:35.951 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T03:28:35.951 INFO:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-11-19T03:28:36.009 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:28:36.010 INFO:tasks.ceph:role: c1.osd.0 2020-11-19T03:28:36.010 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:28:36.010 INFO:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-19T03:28:36.537 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T03:28:36.537 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T03:28:36.537 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T03:28:36.537 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T03:28:36.538 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2020-11-19T03:28:36.538 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T03:28:36.538 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T03:28:36.538 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T03:28:36.538 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T03:28:36.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2020-11-19T03:28:36.540 INFO:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-11-19T03:28:36.622 INFO:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-11-19T03:28:36.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T03:28:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T03:28:36.639 INFO:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-11-19T03:28:36.700 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:28:36.701 INFO:tasks.ceph:role: c1.osd.1 2020-11-19T03:28:36.701 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:28:36.703 INFO:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-19T03:28:37.221 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T03:28:37.222 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T03:28:37.222 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T03:28:37.222 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T03:28:37.222 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2020-11-19T03:28:37.223 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T03:28:37.223 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T03:28:37.223 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T03:28:37.223 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T03:28:37.224 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2020-11-19T03:28:37.225 INFO:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-11-19T03:28:37.269 INFO:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-11-19T03:28:37.325 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T03:28:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T03:28:37.327 INFO:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-11-19T03:28:37.384 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:28:37.385 INFO:tasks.ceph:role: c1.osd.2 2020-11-19T03:28:37.385 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:28:37.386 INFO:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-19T03:28:37.919 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T03:28:37.920 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T03:28:37.920 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T03:28:37.921 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T03:28:37.921 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2020-11-19T03:28:37.921 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T03:28:37.922 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T03:28:37.922 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T03:28:37.923 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T03:28:37.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2020-11-19T03:28:37.924 INFO:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-11-19T03:28:37.967 INFO:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-11-19T03:28:38.024 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T03:28:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T03:28:38.026 INFO:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:38.114 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:38.108+0000 7f5061e97d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-11-19T03:28:38.115 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:38.108+0000 7f5061e97d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-11-19T03:28:38.115 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:38.108+0000 7f5061e97d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-19T03:28:38.115 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:38.108+0000 7f5061e97d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-19T03:28:38.116 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:38.108+0000 7f5061e97d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-19T03:28:38.116 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:38.108+0000 7f5061e97d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-11-19T03:28:40.282 INFO:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-11-19T03:28:40.299 INFO:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:40.384 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:40.376+0000 7efef3af1d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-11-19T03:28:40.385 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:40.376+0000 7efef3af1d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-11-19T03:28:40.385 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:40.376+0000 7efef3af1d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-19T03:28:40.385 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:40.376+0000 7efef3af1d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-19T03:28:40.385 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:40.376+0000 7efef3af1d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-19T03:28:40.386 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:40.376+0000 7efef3af1d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-11-19T03:28:42.550 INFO:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-11-19T03:28:42.610 INFO:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:42.692 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:42.684+0000 7f8ea68a3d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-11-19T03:28:42.693 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:42.684+0000 7f8ea68a3d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-11-19T03:28:42.693 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:42.684+0000 7f8ea68a3d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-19T03:28:42.693 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:42.684+0000 7f8ea68a3d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-19T03:28:42.693 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:42.684+0000 7f8ea68a3d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-19T03:28:42.694 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:28:42.684+0000 7f8ea68a3d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-11-19T03:28:44.869 INFO:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-11-19T03:28:45.774 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-19T03:28:45.775 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:45.775 INFO:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-19T03:28:45.822 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-19T03:28:45.831 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr.x] 2020-11-19T03:28:45.832 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVfk53vLRAAg4XZwQPkRP8DmY2AHyfB/A== 2020-11-19T03:28:45.832 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:45.832 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:45.832 INFO:teuthology.orchestra.run.smithi066.stderr:56 bytes copied, 0.000170507 s, 328 kB/s 2020-11-19T03:28:45.834 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:45.834 INFO:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-19T03:28:45.883 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-19T03:28:45.892 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.0] 2020-11-19T03:28:45.893 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBm5rVf5VuOBhAAqTymXuRq5joOTPHh4eGJ+Q== 2020-11-19T03:28:45.893 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:45.894 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:45.894 INFO:teuthology.orchestra.run.smithi066.stderr:56 bytes copied, 0.000157297 s, 356 kB/s 2020-11-19T03:28:45.895 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:45.895 INFO:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-19T03:28:45.937 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-19T03:28:45.945 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.1] 2020-11-19T03:28:45.945 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBo5rVfYOeoFhAAv9Y4rDf0G/hvab5+28K4WA== 2020-11-19T03:28:45.946 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:45.946 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:45.946 INFO:teuthology.orchestra.run.smithi066.stderr:56 bytes copied, 0.000162007 s, 346 kB/s 2020-11-19T03:28:45.947 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:45.947 INFO:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-19T03:28:45.994 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-19T03:28:46.003 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.2] 2020-11-19T03:28:46.003 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBq5rVfQbf+KBAASQ//eDxk+o4eXJlhBVykPg== 2020-11-19T03:28:46.003 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:46.003 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:46.004 INFO:teuthology.orchestra.run.smithi066.stderr:56 bytes copied, 0.000167677 s, 334 kB/s 2020-11-19T03:28:46.005 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:46.005 INFO:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-19T03:28:46.050 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-19T03:28:46.051 INFO:teuthology.orchestra.run.smithi066.stdout:[client.0] 2020-11-19T03:28:46.052 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVfzBJzMhAAhDhx1ZNev2jhiPE6PogwqQ== 2020-11-19T03:28:46.052 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:46.052 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:46.052 INFO:teuthology.orchestra.run.smithi066.stderr:59 bytes copied, 0.000169563 s, 348 kB/s 2020-11-19T03:28:46.053 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:46.053 INFO:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-19T03:28:46.098 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-19T03:28:46.099 INFO:teuthology.orchestra.run.smithi066.stdout:[client.1] 2020-11-19T03:28:46.099 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVf2AWyNxAALJoN2nfwACxcOzs0/v6V1w== 2020-11-19T03:28:46.100 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:46.100 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:46.100 INFO:teuthology.orchestra.run.smithi066.stderr:59 bytes copied, 0.000178657 s, 330 kB/s 2020-11-19T03:28:46.100 INFO:tasks.ceph:Adding keys to all mons... 2020-11-19T03:28:46.101 INFO:teuthology.orchestra.run.smithi066:> sudo tee -a /etc/ceph/c1.keyring 2020-11-19T03:28:46.154 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr.x] 2020-11-19T03:28:46.155 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVfk53vLRAAg4XZwQPkRP8DmY2AHyfB/A== 2020-11-19T03:28:46.155 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.0] 2020-11-19T03:28:46.155 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBm5rVf5VuOBhAAqTymXuRq5joOTPHh4eGJ+Q== 2020-11-19T03:28:46.156 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.1] 2020-11-19T03:28:46.156 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBo5rVfYOeoFhAAv9Y4rDf0G/hvab5+28K4WA== 2020-11-19T03:28:46.156 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.2] 2020-11-19T03:28:46.156 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBq5rVfQbf+KBAASQ//eDxk+o4eXJlhBVykPg== 2020-11-19T03:28:46.156 INFO:teuthology.orchestra.run.smithi066.stdout:[client.0] 2020-11-19T03:28:46.157 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVfzBJzMhAAhDhx1ZNev2jhiPE6PogwqQ== 2020-11-19T03:28:46.157 INFO:teuthology.orchestra.run.smithi066.stdout:[client.1] 2020-11-19T03:28:46.157 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBj5rVf2AWyNxAALJoN2nfwACxcOzs0/v6V1w== 2020-11-19T03:28:46.158 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-19T03:28:46.244 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T03:28:46.328 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T03:28:46.412 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T03:28:46.495 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T03:28:46.582 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T03:28:46.674 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-19T03:28:46.675 INFO:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-11-19T03:28:46.733 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-11-19T03:28:46.837 INFO:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-11-19T03:28:46.853 INFO:teuthology.orchestra.run.smithi066:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-11-19T03:28:46.904 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-11-19T03:28:46.904 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-11-19T03:28:46.905 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-11-19T03:28:46.953 INFO:tasks.ceph.c1.mon.a:Started 2020-11-19T03:28:46.954 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-11-19T03:28:46.954 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-11-19T03:28:46.955 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-11-19T03:28:46.960 INFO:tasks.ceph.c1.mgr.x:Started 2020-11-19T03:28:46.960 INFO:tasks.ceph:Setting crush tunables to default 2020-11-19T03:28:46.961 INFO:teuthology.orchestra.run.smithi066:> sudo ceph --cluster c1 osd crush tunables default 2020-11-19T03:28:47.295 INFO:teuthology.orchestra.run.smithi066.stderr:adjusted tunables profile to default 2020-11-19T03:28:47.312 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-11-19T03:28:47.312 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:47.313 INFO:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-19T03:28:47.316 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-19T03:28:47.322 INFO:teuthology.orchestra.run.smithi066.stdout:ec2143e4-d0ea-4014-b813-88ece166b175 2020-11-19T03:28:47.322 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:47.323 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:47.323 INFO:teuthology.orchestra.run.smithi066.stderr:37 bytes copied, 9.6438e-05 s, 384 kB/s 2020-11-19T03:28:47.326 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:47.327 INFO:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-19T03:28:47.374 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-19T03:28:47.382 INFO:teuthology.orchestra.run.smithi066.stdout:a678814e-19e8-459c-91ee-0e8b1451f641 2020-11-19T03:28:47.382 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:47.382 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:47.383 INFO:teuthology.orchestra.run.smithi066.stderr:37 bytes copied, 0.00018724 s, 198 kB/s 2020-11-19T03:28:47.383 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:28:47.384 INFO:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-19T03:28:47.431 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-19T03:28:47.439 INFO:teuthology.orchestra.run.smithi066.stdout:454453c1-e3db-4ebf-a95a-6cd5023971f7 2020-11-19T03:28:47.439 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:28:47.440 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:28:47.440 INFO:teuthology.orchestra.run.smithi066.stderr:37 bytes copied, 0.000163094 s, 227 kB/s 2020-11-19T03:28:47.441 INFO:teuthology.orchestra.run.smithi066:> sudo ceph --cluster c1 osd new ec2143e4-d0ea-4014-b813-88ece166b175 0 2020-11-19T03:28:47.702 INFO:teuthology.orchestra.run.smithi066.stdout:0 2020-11-19T03:28:47.715 INFO:teuthology.orchestra.run.smithi066:> sudo ceph --cluster c1 osd new a678814e-19e8-459c-91ee-0e8b1451f641 1 2020-11-19T03:28:47.935 INFO:teuthology.orchestra.run.smithi066.stdout:1 2020-11-19T03:28:47.951 INFO:teuthology.orchestra.run.smithi066:> sudo ceph --cluster c1 osd new 454453c1-e3db-4ebf-a95a-6cd5023971f7 2 2020-11-19T03:28:48.195 INFO:teuthology.orchestra.run.smithi066.stdout:2 2020-11-19T03:28:48.210 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-11-19T03:28:48.211 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-11-19T03:28:48.214 INFO:tasks.ceph.c1.osd.0:Started 2020-11-19T03:28:48.214 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-11-19T03:28:48.215 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-11-19T03:28:48.218 INFO:tasks.ceph.c1.osd.1:Started 2020-11-19T03:28:48.218 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-11-19T03:28:48.218 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-11-19T03:28:48.222 INFO:tasks.ceph.c1.osd.2:Started 2020-11-19T03:28:48.223 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-19T03:28:48.528 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-19T03:28:48.738 INFO:teuthology.misc.health.smithi066.stdout: 2020-11-19T03:28:50.591 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":5,"fsid":"f9ed144c-5db3-4d83-993d-51016a1aaef3","created":"2020-11-19T03:28:47.084369+0000","modified":"2020-11-19T03:28:48.187659+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"ec2143e4-d0ea-4014-b813-88ece166b175","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a678814e-19e8-459c-91ee-0e8b1451f641","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"454453c1-e3db-4ebf-a95a-6cd5023971f7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:28:50.593 INFO:tasks.ceph.c1.osd.1.smithi066.stderr:2020-11-19T03:28:48.816+0000 7f9859d7cd80 -1 Falling back to public interface 2020-11-19T03:28:50.593 INFO:tasks.ceph.c1.osd.1.smithi066.stderr:2020-11-19T03:28:49.444+0000 7f9859d7cd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-19T03:28:50.593 INFO:tasks.ceph.c1.osd.0.smithi066.stderr:2020-11-19T03:28:48.816+0000 7f0329069d80 -1 Falling back to public interface 2020-11-19T03:28:50.594 INFO:tasks.ceph.c1.osd.0.smithi066.stderr:2020-11-19T03:28:49.436+0000 7f0329069d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-19T03:28:50.594 INFO:tasks.ceph.c1.osd.2.smithi066.stderr:2020-11-19T03:28:48.840+0000 7f4e6c8e2d80 -1 Falling back to public interface 2020-11-19T03:28:50.594 INFO:tasks.ceph.c1.osd.2.smithi066.stderr:2020-11-19T03:28:49.200+0000 7f4e6c8e2d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-19T03:28:50.594 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-19T03:28:51.112 INFO:tasks.ceph.c1.osd.0.smithi066.stderr:2020-11-19T03:28:51.104+0000 7f0320533700 -1 osd.0 0 waiting for initial osdmap 2020-11-19T03:28:51.112 INFO:tasks.ceph.c1.osd.1.smithi066.stderr:2020-11-19T03:28:51.104+0000 7f9851246700 -1 osd.1 0 waiting for initial osdmap 2020-11-19T03:28:51.115 INFO:tasks.ceph.c1.osd.2.smithi066.stderr:2020-11-19T03:28:51.108+0000 7f4e655af700 -1 osd.2 0 waiting for initial osdmap 2020-11-19T03:28:56.899 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-19T03:28:57.194 INFO:teuthology.misc.health.smithi066.stdout: 2020-11-19T03:28:57.194 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"f9ed144c-5db3-4d83-993d-51016a1aaef3","created":"2020-11-19T03:28:47.084369+0000","modified":"2020-11-19T03:28:57.156734+0000","last_up_change":"2020-11-19T03:28:52.107851+0000","last_in_change":"2020-11-19T03:28:52.107851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:28:51.144797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"ec2143e4-d0ea-4014-b813-88ece166b175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":11058},{"type":"v1","addr":"172.21.15.66:6807","nonce":11058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":11058},{"type":"v1","addr":"172.21.15.66:6811","nonce":11058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":11058},{"type":"v1","addr":"172.21.15.66:6815","nonce":11058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":11058},{"type":"v1","addr":"172.21.15.66:6813","nonce":11058}]},"public_addr":"172.21.15.66:6807/11058","cluster_addr":"172.21.15.66:6811/11058","heartbeat_back_addr":"172.21.15.66:6815/11058","heartbeat_front_addr":"172.21.15.66:6813/11058","state":["exists","up"]},{"osd":1,"uuid":"a678814e-19e8-459c-91ee-0e8b1451f641","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":11057},{"type":"v1","addr":"172.21.15.66:6801","nonce":11057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":11057},{"type":"v1","addr":"172.21.15.66:6803","nonce":11057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":11057},{"type":"v1","addr":"172.21.15.66:6810","nonce":11057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":11057},{"type":"v1","addr":"172.21.15.66:6806","nonce":11057}]},"public_addr":"172.21.15.66:6801/11057","cluster_addr":"172.21.15.66:6803/11057","heartbeat_back_addr":"172.21.15.66:6810/11057","heartbeat_front_addr":"172.21.15.66:6806/11057","state":["exists","up"]},{"osd":2,"uuid":"454453c1-e3db-4ebf-a95a-6cd5023971f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":11059},{"type":"v1","addr":"172.21.15.66:6817","nonce":11059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":11059},{"type":"v1","addr":"172.21.15.66:6819","nonce":11059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":11059},{"type":"v1","addr":"172.21.15.66:6823","nonce":11059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":11059},{"type":"v1","addr":"172.21.15.66:6821","nonce":11059}]},"public_addr":"172.21.15.66:6817/11059","cluster_addr":"172.21.15.66:6819/11059","heartbeat_back_addr":"172.21.15.66:6823/11059","heartbeat_front_addr":"172.21.15.66:6821/11059","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.454516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.415797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:28:57.214 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-19T03:28:57.215 INFO:tasks.ceph:Creating RBD pool 2020-11-19T03:28:57.215 INFO:teuthology.orchestra.run.smithi066:> sudo ceph --cluster c1 osd pool create rbd 8 2020-11-19T03:28:58.183 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'rbd' created 2020-11-19T03:28:58.194 INFO:teuthology.orchestra.run.smithi066:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-19T03:28:59.195 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-19T03:28:59.208 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-11-19T03:28:59.209 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-19T03:28:59.209 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-19T03:28:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:28:59.502 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":13,"fsid":"f9ed144c-5db3-4d83-993d-51016a1aaef3","created":"2020-11-19T03:28:47.084369+0000","modified":"2020-11-19T03:28:59.167518+0000","last_up_change":"2020-11-19T03:28:52.107851+0000","last_in_change":"2020-11-19T03:28:52.107851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:28:51.144797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T03:28:57.502216+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ec2143e4-d0ea-4014-b813-88ece166b175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":11058},{"type":"v1","addr":"172.21.15.66:6807","nonce":11058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":11058},{"type":"v1","addr":"172.21.15.66:6811","nonce":11058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":11058},{"type":"v1","addr":"172.21.15.66:6815","nonce":11058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":11058},{"type":"v1","addr":"172.21.15.66:6813","nonce":11058}]},"public_addr":"172.21.15.66:6807/11058","cluster_addr":"172.21.15.66:6811/11058","heartbeat_back_addr":"172.21.15.66:6815/11058","heartbeat_front_addr":"172.21.15.66:6813/11058","state":["exists","up"]},{"osd":1,"uuid":"a678814e-19e8-459c-91ee-0e8b1451f641","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":11057},{"type":"v1","addr":"172.21.15.66:6801","nonce":11057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":11057},{"type":"v1","addr":"172.21.15.66:6803","nonce":11057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":11057},{"type":"v1","addr":"172.21.15.66:6810","nonce":11057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":11057},{"type":"v1","addr":"172.21.15.66:6806","nonce":11057}]},"public_addr":"172.21.15.66:6801/11057","cluster_addr":"172.21.15.66:6803/11057","heartbeat_back_addr":"172.21.15.66:6810/11057","heartbeat_front_addr":"172.21.15.66:6806/11057","state":["exists","up"]},{"osd":2,"uuid":"454453c1-e3db-4ebf-a95a-6cd5023971f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":11059},{"type":"v1","addr":"172.21.15.66:6817","nonce":11059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":11059},{"type":"v1","addr":"172.21.15.66:6819","nonce":11059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":11059},{"type":"v1","addr":"172.21.15.66:6823","nonce":11059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":11059},{"type":"v1","addr":"172.21.15.66:6821","nonce":11059}]},"public_addr":"172.21.15.66:6817/11059","cluster_addr":"172.21.15.66:6819/11059","heartbeat_back_addr":"172.21.15.66:6823/11059","heartbeat_front_addr":"172.21.15.66:6821/11059","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.454516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.415797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.251523+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:28:59.515 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-19T03:28:51.144797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-19T03:28:57.502216+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-19T03:28:59.516 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2020-11-19T03:28:59.823 INFO:teuthology.orchestra.run.smithi066.stdout:pg_num: 1 2020-11-19T03:28:59.838 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2020-11-19T03:29:00.136 INFO:teuthology.orchestra.run.smithi066.stdout:pg_num: 8 2020-11-19T03:29:00.150 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-11-19T03:29:00.150 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-11-19T03:29:00.151 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2020-11-19T03:29:00.478 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:00.492 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6824","nonce":10873},{"type":"v1","addr":"172.21.15.66:6825","nonce":10873}]},"active_addr":"172.21.15.66:6825/10873","active_change":"2020-11-19T03:28:51.122675+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":255202538}]},{"addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":1755741480}]},{"addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3602552285}]}]}} 2020-11-19T03:29:00.495 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-11-19T03:29:00.495 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-11-19T03:29:00.495 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-19T03:29:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:00.792 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":14,"fsid":"f9ed144c-5db3-4d83-993d-51016a1aaef3","created":"2020-11-19T03:28:47.084369+0000","modified":"2020-11-19T03:29:00.171714+0000","last_up_change":"2020-11-19T03:28:52.107851+0000","last_in_change":"2020-11-19T03:28:52.107851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:28:51.144797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T03:28:57.502216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ec2143e4-d0ea-4014-b813-88ece166b175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":11058},{"type":"v1","addr":"172.21.15.66:6807","nonce":11058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":11058},{"type":"v1","addr":"172.21.15.66:6811","nonce":11058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":11058},{"type":"v1","addr":"172.21.15.66:6815","nonce":11058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":11058},{"type":"v1","addr":"172.21.15.66:6813","nonce":11058}]},"public_addr":"172.21.15.66:6807/11058","cluster_addr":"172.21.15.66:6811/11058","heartbeat_back_addr":"172.21.15.66:6815/11058","heartbeat_front_addr":"172.21.15.66:6813/11058","state":["exists","up"]},{"osd":1,"uuid":"a678814e-19e8-459c-91ee-0e8b1451f641","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":11057},{"type":"v1","addr":"172.21.15.66:6801","nonce":11057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":11057},{"type":"v1","addr":"172.21.15.66:6803","nonce":11057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":11057},{"type":"v1","addr":"172.21.15.66:6810","nonce":11057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":11057},{"type":"v1","addr":"172.21.15.66:6806","nonce":11057}]},"public_addr":"172.21.15.66:6801/11057","cluster_addr":"172.21.15.66:6803/11057","heartbeat_back_addr":"172.21.15.66:6810/11057","heartbeat_front_addr":"172.21.15.66:6806/11057","state":["exists","up"]},{"osd":2,"uuid":"454453c1-e3db-4ebf-a95a-6cd5023971f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":11059},{"type":"v1","addr":"172.21.15.66:6817","nonce":11059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":11059},{"type":"v1","addr":"172.21.15.66:6819","nonce":11059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":11059},{"type":"v1","addr":"172.21.15.66:6823","nonce":11059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":11059},{"type":"v1","addr":"172.21.15.66:6821","nonce":11059}]},"public_addr":"172.21.15.66:6817/11059","cluster_addr":"172.21.15.66:6819/11059","heartbeat_back_addr":"172.21.15.66:6823/11059","heartbeat_front_addr":"172.21.15.66:6821/11059","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.454516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.415797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.251523+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:29:00.807 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-11-19T03:29:00.807 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-19T03:29:01.109 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:01.109 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":14,"fsid":"f9ed144c-5db3-4d83-993d-51016a1aaef3","created":"2020-11-19T03:28:47.084369+0000","modified":"2020-11-19T03:29:00.171714+0000","last_up_change":"2020-11-19T03:28:52.107851+0000","last_in_change":"2020-11-19T03:28:52.107851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:28:51.144797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T03:28:57.502216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ec2143e4-d0ea-4014-b813-88ece166b175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":11058},{"type":"v1","addr":"172.21.15.66:6807","nonce":11058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":11058},{"type":"v1","addr":"172.21.15.66:6811","nonce":11058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6814","nonce":11058},{"type":"v1","addr":"172.21.15.66:6815","nonce":11058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6812","nonce":11058},{"type":"v1","addr":"172.21.15.66:6813","nonce":11058}]},"public_addr":"172.21.15.66:6807/11058","cluster_addr":"172.21.15.66:6811/11058","heartbeat_back_addr":"172.21.15.66:6815/11058","heartbeat_front_addr":"172.21.15.66:6813/11058","state":["exists","up"]},{"osd":1,"uuid":"a678814e-19e8-459c-91ee-0e8b1451f641","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":11057},{"type":"v1","addr":"172.21.15.66:6801","nonce":11057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":11057},{"type":"v1","addr":"172.21.15.66:6803","nonce":11057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":11057},{"type":"v1","addr":"172.21.15.66:6810","nonce":11057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":11057},{"type":"v1","addr":"172.21.15.66:6806","nonce":11057}]},"public_addr":"172.21.15.66:6801/11057","cluster_addr":"172.21.15.66:6803/11057","heartbeat_back_addr":"172.21.15.66:6810/11057","heartbeat_front_addr":"172.21.15.66:6806/11057","state":["exists","up"]},{"osd":2,"uuid":"454453c1-e3db-4ebf-a95a-6cd5023971f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6816","nonce":11059},{"type":"v1","addr":"172.21.15.66:6817","nonce":11059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6818","nonce":11059},{"type":"v1","addr":"172.21.15.66:6819","nonce":11059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6822","nonce":11059},{"type":"v1","addr":"172.21.15.66:6823","nonce":11059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6820","nonce":11059},{"type":"v1","addr":"172.21.15.66:6821","nonce":11059}]},"public_addr":"172.21.15.66:6817/11059","cluster_addr":"172.21.15.66:6819/11059","heartbeat_back_addr":"172.21.15.66:6823/11059","heartbeat_front_addr":"172.21.15.66:6821/11059","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.454516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.415797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:28:50.251523+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:29:01.125 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2020-11-19T03:29:01.292 INFO:teuthology.orchestra.run.smithi066.stdout:34359738371 2020-11-19T03:29:01.293 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2020-11-19T03:29:01.460 INFO:teuthology.orchestra.run.smithi066.stdout:34359738371 2020-11-19T03:29:01.460 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2020-11-19T03:29:01.633 INFO:teuthology.orchestra.run.smithi066.stdout:34359738371 2020-11-19T03:29:01.634 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-19T03:29:01.942 INFO:teuthology.orchestra.run.smithi066.stdout:34359738370 2020-11-19T03:29:01.957 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-11-19T03:29:02.959 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-19T03:29:03.258 INFO:teuthology.orchestra.run.smithi066.stdout:34359738372 2020-11-19T03:29:03.273 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.0 2020-11-19T03:29:03.274 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2020-11-19T03:29:03.570 INFO:teuthology.orchestra.run.smithi066.stdout:34359738372 2020-11-19T03:29:03.584 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.1 2020-11-19T03:29:03.584 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2020-11-19T03:29:03.876 INFO:teuthology.orchestra.run.smithi066.stdout:34359738372 2020-11-19T03:29:03.891 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.2 2020-11-19T03:29:03.891 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-11-19T03:29:03.891 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-19T03:29:04.182 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:04.183 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2020-11-19T03:29:04.193 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-11-19T03:29:03.129269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14768,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558032,"statfs":{"total":322122547200,"available":322107424768,"internally_reserved":0,"allocated":352256,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.970000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188767+0000","last_change":"2020-11-19T03:28:59.187424+0000","last_active":"2020-11-19T03:28:59.188767+0000","last_peered":"2020-11-19T03:28:59.188767+0000","last_clean":"2020-11-19T03:28:59.188767+0000","last_became_active":"2020-11-19T03:28:59.186601+0000","last_became_peered":"2020-11-19T03:28:59.186601+0000","last_unstale":"2020-11-19T03:28:59.188767+0000","last_undegraded":"2020-11-19T03:28:59.188767+0000","last_fullsized":"2020-11-19T03:28:59.188767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188171+0000","last_change":"2020-11-19T03:28:59.186112+0000","last_active":"2020-11-19T03:28:59.188171+0000","last_peered":"2020-11-19T03:28:59.188171+0000","last_clean":"2020-11-19T03:28:59.188171+0000","last_became_active":"2020-11-19T03:28:59.185400+0000","last_became_peered":"2020-11-19T03:28:59.185400+0000","last_unstale":"2020-11-19T03:28:59.188171+0000","last_undegraded":"2020-11-19T03:28:59.188171+0000","last_fullsized":"2020-11-19T03:28:59.188171+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188730+0000","last_change":"2020-11-19T03:28:59.187315+0000","last_active":"2020-11-19T03:28:59.188730+0000","last_peered":"2020-11-19T03:28:59.188730+0000","last_clean":"2020-11-19T03:28:59.188730+0000","last_became_active":"2020-11-19T03:28:59.186812+0000","last_became_peered":"2020-11-19T03:28:59.186812+0000","last_unstale":"2020-11-19T03:28:59.188730+0000","last_undegraded":"2020-11-19T03:28:59.188730+0000","last_fullsized":"2020-11-19T03:28:59.188730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188607+0000","last_change":"2020-11-19T03:28:59.188607+0000","last_active":"2020-11-19T03:28:59.188607+0000","last_peered":"2020-11-19T03:28:59.188607+0000","last_clean":"2020-11-19T03:28:59.188607+0000","last_became_active":"2020-11-19T03:28:59.187594+0000","last_became_peered":"2020-11-19T03:28:59.187594+0000","last_unstale":"2020-11-19T03:28:59.188607+0000","last_undegraded":"2020-11-19T03:28:59.188607+0000","last_fullsized":"2020-11-19T03:28:59.188607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"14","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.176144+0000","last_change":"2020-11-19T03:28:53.510556+0000","last_active":"2020-11-19T03:28:59.176144+0000","last_peered":"2020-11-19T03:28:59.176144+0000","last_clean":"2020-11-19T03:28:59.176144+0000","last_became_active":"2020-11-19T03:28:53.508700+0000","last_became_peered":"2020-11-19T03:28:53.508700+0000","last_unstale":"2020-11-19T03:28:59.176144+0000","last_undegraded":"2020-11-19T03:28:59.176144+0000","last_fullsized":"2020-11-19T03:28:59.176144+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:52.107851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:52.107851+0000","last_clean_scrub_stamp":"2020-11-19T03:28:52.107851+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.187435+0000","last_change":"2020-11-19T03:28:59.185420+0000","last_active":"2020-11-19T03:28:59.187435+0000","last_peered":"2020-11-19T03:28:59.187435+0000","last_clean":"2020-11-19T03:28:59.187435+0000","last_became_active":"2020-11-19T03:28:59.184753+0000","last_became_peered":"2020-11-19T03:28:59.184753+0000","last_unstale":"2020-11-19T03:28:59.187435+0000","last_undegraded":"2020-11-19T03:28:59.187435+0000","last_fullsized":"2020-11-19T03:28:59.187435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.190123+0000","last_change":"2020-11-19T03:28:59.190123+0000","last_active":"2020-11-19T03:28:59.190123+0000","last_peered":"2020-11-19T03:28:59.190123+0000","last_clean":"2020-11-19T03:28:59.190123+0000","last_became_active":"2020-11-19T03:28:59.189702+0000","last_became_peered":"2020-11-19T03:28:59.189702+0000","last_unstale":"2020-11-19T03:28:59.190123+0000","last_undegraded":"2020-11-19T03:28:59.190123+0000","last_fullsized":"2020-11-19T03:28:59.190123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.189685+0000","last_change":"2020-11-19T03:28:59.189685+0000","last_active":"2020-11-19T03:28:59.189685+0000","last_peered":"2020-11-19T03:28:59.189685+0000","last_clean":"2020-11-19T03:28:59.189685+0000","last_became_active":"2020-11-19T03:28:59.188885+0000","last_became_peered":"2020-11-19T03:28:59.188885+0000","last_unstale":"2020-11-19T03:28:59.189685+0000","last_undegraded":"2020-11-19T03:28:59.189685+0000","last_fullsized":"2020-11-19T03:28:59.189685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188800+0000","last_change":"2020-11-19T03:28:59.187672+0000","last_active":"2020-11-19T03:28:59.188800+0000","last_peered":"2020-11-19T03:28:59.188800+0000","last_clean":"2020-11-19T03:28:59.188800+0000","last_became_active":"2020-11-19T03:28:59.186914+0000","last_became_peered":"2020-11-19T03:28:59.186914+0000","last_unstale":"2020-11-19T03:28:59.188800+0000","last_undegraded":"2020-11-19T03:28:59.188800+0000","last_fullsized":"2020-11-19T03:28:59.188800+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":35558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]}]}]}} 2020-11-19T03:29:04.194 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-19T03:29:04.487 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:04.488 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2020-11-19T03:29:04.501 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-11-19T03:29:03.129269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14768,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558032,"statfs":{"total":322122547200,"available":322107424768,"internally_reserved":0,"allocated":352256,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.970000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188767+0000","last_change":"2020-11-19T03:28:59.187424+0000","last_active":"2020-11-19T03:28:59.188767+0000","last_peered":"2020-11-19T03:28:59.188767+0000","last_clean":"2020-11-19T03:28:59.188767+0000","last_became_active":"2020-11-19T03:28:59.186601+0000","last_became_peered":"2020-11-19T03:28:59.186601+0000","last_unstale":"2020-11-19T03:28:59.188767+0000","last_undegraded":"2020-11-19T03:28:59.188767+0000","last_fullsized":"2020-11-19T03:28:59.188767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188171+0000","last_change":"2020-11-19T03:28:59.186112+0000","last_active":"2020-11-19T03:28:59.188171+0000","last_peered":"2020-11-19T03:28:59.188171+0000","last_clean":"2020-11-19T03:28:59.188171+0000","last_became_active":"2020-11-19T03:28:59.185400+0000","last_became_peered":"2020-11-19T03:28:59.185400+0000","last_unstale":"2020-11-19T03:28:59.188171+0000","last_undegraded":"2020-11-19T03:28:59.188171+0000","last_fullsized":"2020-11-19T03:28:59.188171+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188730+0000","last_change":"2020-11-19T03:28:59.187315+0000","last_active":"2020-11-19T03:28:59.188730+0000","last_peered":"2020-11-19T03:28:59.188730+0000","last_clean":"2020-11-19T03:28:59.188730+0000","last_became_active":"2020-11-19T03:28:59.186812+0000","last_became_peered":"2020-11-19T03:28:59.186812+0000","last_unstale":"2020-11-19T03:28:59.188730+0000","last_undegraded":"2020-11-19T03:28:59.188730+0000","last_fullsized":"2020-11-19T03:28:59.188730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188607+0000","last_change":"2020-11-19T03:28:59.188607+0000","last_active":"2020-11-19T03:28:59.188607+0000","last_peered":"2020-11-19T03:28:59.188607+0000","last_clean":"2020-11-19T03:28:59.188607+0000","last_became_active":"2020-11-19T03:28:59.187594+0000","last_became_peered":"2020-11-19T03:28:59.187594+0000","last_unstale":"2020-11-19T03:28:59.188607+0000","last_undegraded":"2020-11-19T03:28:59.188607+0000","last_fullsized":"2020-11-19T03:28:59.188607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"14","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.176144+0000","last_change":"2020-11-19T03:28:53.510556+0000","last_active":"2020-11-19T03:28:59.176144+0000","last_peered":"2020-11-19T03:28:59.176144+0000","last_clean":"2020-11-19T03:28:59.176144+0000","last_became_active":"2020-11-19T03:28:53.508700+0000","last_became_peered":"2020-11-19T03:28:53.508700+0000","last_unstale":"2020-11-19T03:28:59.176144+0000","last_undegraded":"2020-11-19T03:28:59.176144+0000","last_fullsized":"2020-11-19T03:28:59.176144+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:52.107851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:52.107851+0000","last_clean_scrub_stamp":"2020-11-19T03:28:52.107851+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.187435+0000","last_change":"2020-11-19T03:28:59.185420+0000","last_active":"2020-11-19T03:28:59.187435+0000","last_peered":"2020-11-19T03:28:59.187435+0000","last_clean":"2020-11-19T03:28:59.187435+0000","last_became_active":"2020-11-19T03:28:59.184753+0000","last_became_peered":"2020-11-19T03:28:59.184753+0000","last_unstale":"2020-11-19T03:28:59.187435+0000","last_undegraded":"2020-11-19T03:28:59.187435+0000","last_fullsized":"2020-11-19T03:28:59.187435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.190123+0000","last_change":"2020-11-19T03:28:59.190123+0000","last_active":"2020-11-19T03:28:59.190123+0000","last_peered":"2020-11-19T03:28:59.190123+0000","last_clean":"2020-11-19T03:28:59.190123+0000","last_became_active":"2020-11-19T03:28:59.189702+0000","last_became_peered":"2020-11-19T03:28:59.189702+0000","last_unstale":"2020-11-19T03:28:59.190123+0000","last_undegraded":"2020-11-19T03:28:59.190123+0000","last_fullsized":"2020-11-19T03:28:59.190123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.189685+0000","last_change":"2020-11-19T03:28:59.189685+0000","last_active":"2020-11-19T03:28:59.189685+0000","last_peered":"2020-11-19T03:28:59.189685+0000","last_clean":"2020-11-19T03:28:59.189685+0000","last_became_active":"2020-11-19T03:28:59.188885+0000","last_became_peered":"2020-11-19T03:28:59.188885+0000","last_unstale":"2020-11-19T03:28:59.189685+0000","last_undegraded":"2020-11-19T03:28:59.189685+0000","last_fullsized":"2020-11-19T03:28:59.189685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2020-11-19T03:28:59.188800+0000","last_change":"2020-11-19T03:28:59.187672+0000","last_active":"2020-11-19T03:28:59.188800+0000","last_peered":"2020-11-19T03:28:59.188800+0000","last_clean":"2020-11-19T03:28:59.188800+0000","last_became_active":"2020-11-19T03:28:59.186914+0000","last_became_peered":"2020-11-19T03:28:59.186914+0000","last_unstale":"2020-11-19T03:28:59.188800+0000","last_undegraded":"2020-11-19T03:28:59.188800+0000","last_fullsized":"2020-11-19T03:28:59.188800+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:28:58.158089+0000","last_clean_scrub_stamp":"2020-11-19T03:28:58.158089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":35558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]}]}]}} 2020-11-19T03:29:04.502 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-11-19T03:29:04.503 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-11-19T03:29:04.503 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-11-19T03:29:04.503 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2020-11-19T03:29:04.840 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:04.856 INFO:teuthology.orchestra.run.smithi066.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-19T03:29:04.857 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-11-19T03:29:04.858 INFO:teuthology.run_tasks:Running task ceph... 2020-11-19T03:29:04.880 INFO:tasks.ceph:Creating ceph cluster c2... 2020-11-19T03:29:04.881 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-19T03:29:04.882 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi/5633886', 'branch': 'wip-yuval-fix-47904', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5633886', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-11-17_16:32:29-rgw-wip-yuval-fix-47904-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuval-fix-47904'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuval-fix-47904', 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuval-fix-47904', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-fix-47904/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '409eac507e130adcc9fee71aa5fd16ed7f289313', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.1594', 'targets': {'smithi066.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcRZ1FksMeIk6CFCb3W6w2tfikSMp9EDkwlSGqUYBi2RLjZJn2SE1rITizpE4X5hGFzRU8SdaWIy3iuhD4VIPrEIa7/SFpk9OfJzkorD6I72swHFJKBahLLH6aJX5Yx8/I7BLiYXzJ0+xyEfobvaZZVf/7gyHldHTDYOU9Lls45PF0S/RN/ZXE+S16ME6+Ef0pIfkwBENcFDcKBdFcL5pk0z6jedB7GMwwrXsGd6EBcOUYxhR6y+wxkMgHMSBVyflVwTK85snOI6shbg9PrvqdnZRkv24i1Kc2lE33YwhXgnp1lLhSceER2/FXlgmz2th2cIsnFOCDY7u12H8T+knj', 'smithi154.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5EGCCacrU4DTfSf3En71+Jcujq6iYhrNBfANMGFlLcrKHyhaRuoNs89UKHMHPSgyJL+hi/5JStDf/DpnvV3aXcLCOL414b4ohNF0Gfrc6PGDmt3WNAjLNr3pweEUgkYpyRFH4f5snFluElHQJ+KdwuU6GAS/c/a87rPkLUAyCa0LaxR5x3dMC+MXH7nRmaOITXjAQNO8e5u1AZPubTFb73nWWHmeBD3tSav1CVI8x1zYgVIfIDrHTeLQwy4dviozplDeUZP/GL0yvq2RxTEbUdWhytvVgnEyCCcuyXmAQaaCc+K8k0Rsw3UqUnhNr1XV/dqaxHFKeW61wcgo1B8GR'}} 2020-11-19T03:29:04.884 INFO:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-19T03:29:04.890 INFO:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-19T03:29:04.903 INFO:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T03:29:04.944 INFO:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2020-11-19T03:29:04.966 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:04.967 INFO:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2020-11-19T03:29:05.014 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-19T03:29:05.015 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_1 2020-11-19T03:29:05.016 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_2 2020-11-19T03:29:05.016 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_3 2020-11-19T03:29:05.016 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_4 2020-11-19T03:29:05.016 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:05.017 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:05.017 INFO:teuthology.orchestra.run.smithi154.stderr:72 bytes copied, 0.000167257 s, 430 kB/s 2020-11-19T03:29:05.017 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-19T03:29:05.020 INFO:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-19T03:29:05.069 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-19T03:29:05.069 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:29:05.069 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 342 Links: 1 2020-11-19T03:29:05.070 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:29:05.070 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2020-11-19 03:26:06.714241983 +0000 2020-11-19T03:29:05.070 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2020-11-19 03:26:06.602241980 +0000 2020-11-19T03:29:05.070 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2020-11-19 03:26:06.602241980 +0000 2020-11-19T03:29:05.071 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2020-11-19T03:29:05.079 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2020-11-19T03:29:05.079 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2020-11-19T03:29:05.079 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000364184 s, 1.4 MB/s 2020-11-19T03:29:05.084 INFO:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-19T03:29:05.136 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-19T03:29:05.136 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:29:05.137 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 349 Links: 1 2020-11-19T03:29:05.137 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:29:05.138 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2020-11-19 03:26:06.874241988 +0000 2020-11-19T03:29:05.138 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2020-11-19 03:26:06.870241988 +0000 2020-11-19T03:29:05.138 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2020-11-19 03:26:06.870241988 +0000 2020-11-19T03:29:05.139 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2020-11-19T03:29:05.145 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2020-11-19T03:29:05.146 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2020-11-19T03:29:05.146 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000350154 s, 1.5 MB/s 2020-11-19T03:29:05.150 INFO:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-19T03:29:05.205 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-19T03:29:05.207 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:29:05.207 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 355 Links: 1 2020-11-19T03:29:05.207 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:29:05.208 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2020-11-19 03:26:07.194241999 +0000 2020-11-19T03:29:05.208 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2020-11-19 03:26:07.106241996 +0000 2020-11-19T03:29:05.208 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2020-11-19 03:26:07.106241996 +0000 2020-11-19T03:29:05.208 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2020-11-19T03:29:05.214 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2020-11-19T03:29:05.214 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2020-11-19T03:29:05.215 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000373628 s, 1.4 MB/s 2020-11-19T03:29:05.218 INFO:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-19T03:29:05.269 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-19T03:29:05.270 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-19T03:29:05.270 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 361 Links: 1 2020-11-19T03:29:05.271 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-19T03:29:05.271 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2020-11-19 03:26:07.450242007 +0000 2020-11-19T03:29:05.271 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2020-11-19 03:26:07.322242003 +0000 2020-11-19T03:29:05.271 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2020-11-19 03:26:07.322242003 +0000 2020-11-19T03:29:05.272 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2020-11-19T03:29:05.279 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2020-11-19T03:29:05.280 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2020-11-19T03:29:05.280 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000408197 s, 1.3 MB/s 2020-11-19T03:29:05.284 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:29:05.284 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T03:29:05.285 INFO:tasks.ceph:Generating config... 2020-11-19T03:29:05.286 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-19T03:29:05.286 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-19T03:29:05.287 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-19T03:29:05.287 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T03:29:05.288 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-19T03:29:05.288 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-19T03:29:05.288 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-19T03:29:05.289 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-19T03:29:05.289 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-19T03:29:05.289 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-19T03:29:05.290 INFO:tasks.ceph:[client] setuser = ceph 2020-11-19T03:29:05.290 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-19T03:29:05.290 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-19T03:29:05.290 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-19T03:29:05.291 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-19T03:29:05.291 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-19T03:29:05.291 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-19T03:29:05.292 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-19T03:29:05.292 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-19T03:29:05.293 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-19T03:29:05.293 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-19T03:29:05.293 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-19T03:29:05.294 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-19T03:29:05.294 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-19T03:29:05.294 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-19T03:29:05.296 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-19T03:29:05.297 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-19T03:29:05.297 INFO:tasks.ceph:Setting up c2.mon.a... 2020-11-19T03:29:05.298 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-11-19T03:29:05.356 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/c2.keyring 2020-11-19T03:29:05.358 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-11-19T03:29:05.435 INFO:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T03:29:05.489 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.154')] 2020-11-19T03:29:05.490 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.154', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-19T03:29:05.491 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:05.491 INFO:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T03:29:05.537 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T03:29:05.538 INFO:teuthology.orchestra.run.smithi154.stderr:6+1 records in 2020-11-19T03:29:05.539 INFO:teuthology.orchestra.run.smithi154.stderr:6+1 records out 2020-11-19T03:29:05.539 INFO:teuthology.orchestra.run.smithi154.stderr:3136 bytes (3.1 kB, 3.1 KiB) copied, 0.000115954 s, 27.0 MB/s 2020-11-19T03:29:05.539 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.154 --print /home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:05.610 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:29:05.610 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:29:05.611 INFO:teuthology.orchestra.run.smithi154.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:05.611 INFO:teuthology.orchestra.run.smithi154.stdout:monmaptool: generated fsid d81238a5-f5eb-4f92-b40e-e63250eb12be 2020-11-19T03:29:05.611 INFO:teuthology.orchestra.run.smithi154.stdout:epoch 0 2020-11-19T03:29:05.611 INFO:teuthology.orchestra.run.smithi154.stdout:fsid d81238a5-f5eb-4f92-b40e-e63250eb12be 2020-11-19T03:29:05.612 INFO:teuthology.orchestra.run.smithi154.stdout:last_changed 2020-11-19T03:29:05.609433+0000 2020-11-19T03:29:05.612 INFO:teuthology.orchestra.run.smithi154.stdout:created 2020-11-19T03:29:05.609433+0000 2020-11-19T03:29:05.612 INFO:teuthology.orchestra.run.smithi154.stdout:min_mon_release 0 (unknown) 2020-11-19T03:29:05.612 INFO:teuthology.orchestra.run.smithi154.stdout:election_strategy: 1 2020-11-19T03:29:05.612 INFO:teuthology.orchestra.run.smithi154.stdout:0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2020-11-19T03:29:05.613 INFO:teuthology.orchestra.run.smithi154.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-11-19T03:29:05.613 INFO:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-19T03:29:05.662 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID d81238a5-f5eb-4f92-b40e-e63250eb12be... 2020-11-19T03:29:05.663 INFO:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-19T03:29:05.666 INFO:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-19T03:29:05.707 INFO:teuthology.orchestra.run.smithi066.stdout:[global] 2020-11-19T03:29:05.707 INFO:teuthology.orchestra.run.smithi066.stdout: chdir = "" 2020-11-19T03:29:05.708 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T03:29:05.708 INFO:teuthology.orchestra.run.smithi066.stdout: auth supported = cephx 2020-11-19T03:29:05.708 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.708 INFO:teuthology.orchestra.run.smithi066.stdout: filestore xattr use omap = true 2020-11-19T03:29:05.708 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.709 INFO:teuthology.orchestra.run.smithi066.stdout: mon clock drift allowed = 1.000 2020-11-19T03:29:05.709 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.709 INFO:teuthology.orchestra.run.smithi066.stdout: osd crush chooseleaf type = 0 2020-11-19T03:29:05.709 INFO:teuthology.orchestra.run.smithi066.stdout: auth debug = true 2020-11-19T03:29:05.709 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.710 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on old message = true 2020-11-19T03:29:05.710 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on bug = true 2020-11-19T03:29:05.710 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.710 INFO:teuthology.orchestra.run.smithi066.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T03:29:05.710 INFO:teuthology.orchestra.run.smithi066.stdout: mon pg warn max object skew = 0 2020-11-19T03:29:05.711 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.711 INFO:teuthology.orchestra.run.smithi066.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T03:29:05.711 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T03:29:05.711 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.711 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default size = 2 2020-11-19T03:29:05.711 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.712 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow primary affinity = true 2020-11-19T03:29:05.712 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow pg remap = true 2020-11-19T03:29:05.712 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on legacy crush tunables = false 2020-11-19T03:29:05.712 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on crush straw calc version zero = false 2020-11-19T03:29:05.712 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on no sortbitwise = false 2020-11-19T03:29:05.713 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on osd down out interval zero = false 2020-11-19T03:29:05.713 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on too few osds = false 2020-11-19T03:29:05.713 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T03:29:05.713 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T03:29:05.713 INFO:teuthology.orchestra.run.smithi066.stdout: mon_allow_pool_size_one = true 2020-11-19T03:29:05.714 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.714 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T03:29:05.714 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.714 INFO:teuthology.orchestra.run.smithi066.stdout: osd default data pool replay window = 5 2020-11-19T03:29:05.714 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.715 INFO:teuthology.orchestra.run.smithi066.stdout: mon allow pool delete = true 2020-11-19T03:29:05.715 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.715 INFO:teuthology.orchestra.run.smithi066.stdout: mon cluster log file level = debug 2020-11-19T03:29:05.715 INFO:teuthology.orchestra.run.smithi066.stdout: debug asserts on shutdown = true 2020-11-19T03:29:05.715 INFO:teuthology.orchestra.run.smithi066.stdout: mon health detail to clog = false 2020-11-19T03:29:05.716 INFO:teuthology.orchestra.run.smithi066.stdout: mon host = 172.21.15.154 2020-11-19T03:29:05.716 INFO:teuthology.orchestra.run.smithi066.stdout: osd heartbeat grace = 40 2020-11-19T03:29:05.716 INFO:teuthology.orchestra.run.smithi066.stdout: fsid = d81238a5-f5eb-4f92-b40e-e63250eb12be 2020-11-19T03:29:05.716 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.716 INFO:teuthology.orchestra.run.smithi066.stdout:[osd] 2020-11-19T03:29:05.717 INFO:teuthology.orchestra.run.smithi066.stdout: osd journal size = 100 2020-11-19T03:29:05.717 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.717 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub load threshold = 5.0 2020-11-19T03:29:05.717 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub max interval = 600 2020-11-19T03:29:05.717 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.718 INFO:teuthology.orchestra.run.smithi066.stdout: osd recover clone overlap = true 2020-11-19T03:29:05.718 INFO:teuthology.orchestra.run.smithi066.stdout: osd recovery max chunk = 1048576 2020-11-19T03:29:05.718 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.718 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug shutdown = true 2020-11-19T03:29:05.718 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug op order = true 2020-11-19T03:29:05.719 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify stray on activate = true 2020-11-19T03:29:05.719 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.719 INFO:teuthology.orchestra.run.smithi066.stdout: osd open classes on start = true 2020-11-19T03:29:05.719 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug pg log writeout = true 2020-11-19T03:29:05.719 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.719 INFO:teuthology.orchestra.run.smithi066.stdout: osd deep scrub update digest min age = 30 2020-11-19T03:29:05.720 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.720 INFO:teuthology.orchestra.run.smithi066.stdout: osd map max advance = 10 2020-11-19T03:29:05.720 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.720 INFO:teuthology.orchestra.run.smithi066.stdout: journal zero on create = true 2020-11-19T03:29:05.720 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.721 INFO:teuthology.orchestra.run.smithi066.stdout: filestore ondisk finisher threads = 3 2020-11-19T03:29:05.721 INFO:teuthology.orchestra.run.smithi066.stdout: filestore apply finisher threads = 3 2020-11-19T03:29:05.721 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.721 INFO:teuthology.orchestra.run.smithi066.stdout: bdev debug aio = true 2020-11-19T03:29:05.721 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug misdirected ops = true 2020-11-19T03:29:05.722 INFO:teuthology.orchestra.run.smithi066.stdout: debug filestore = 20 2020-11-19T03:29:05.722 INFO:teuthology.orchestra.run.smithi066.stdout: debug journal = 20 2020-11-19T03:29:05.722 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 20 2020-11-19T03:29:05.722 INFO:teuthology.orchestra.run.smithi066.stdout: debug osd = 25 2020-11-19T03:29:05.722 INFO:teuthology.orchestra.run.smithi066.stdout: osd fast shutdown = False 2020-11-19T03:29:05.723 INFO:teuthology.orchestra.run.smithi066.stdout: rocksdb delete range threshold = 0 2020-11-19T03:29:05.723 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.723 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr] 2020-11-19T03:29:05.723 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2020-11-19T03:29:05.723 INFO:teuthology.orchestra.run.smithi066.stdout: debug mgr = 20 2020-11-19T03:29:05.724 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2020-11-19T03:29:05.724 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2020-11-19T03:29:05.724 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:29:05.724 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:29:05.724 INFO:teuthology.orchestra.run.smithi066.stdout: mgr/telemetry/nag = false 2020-11-19T03:29:05.724 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.725 INFO:teuthology.orchestra.run.smithi066.stdout:[mon] 2020-11-19T03:29:05.725 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2020-11-19T03:29:05.725 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2020-11-19T03:29:05.725 INFO:teuthology.orchestra.run.smithi066.stdout: debug paxos = 20 2020-11-19T03:29:05.725 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2020-11-19T03:29:05.726 INFO:teuthology.orchestra.run.smithi066.stdout: mon data avail warn = 5 2020-11-19T03:29:05.726 INFO:teuthology.orchestra.run.smithi066.stdout: mon mgr mkfs grace = 240 2020-11-19T03:29:05.726 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:29:05.726 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd reporter subtree level = osd 2020-11-19T03:29:05.726 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd prime pg temp = true 2020-11-19T03:29:05.727 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:29:05.727 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd crush smoke test = False 2020-11-19T03:29:05.727 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:29:05.727 INFO:teuthology.orchestra.run.smithi066.stdout:[client] 2020-11-19T03:29:05.727 INFO:teuthology.orchestra.run.smithi066.stdout: rgw cache enabled = true 2020-11-19T03:29:05.728 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable ops log = true 2020-11-19T03:29:05.728 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable usage log = true 2020-11-19T03:29:05.728 INFO:teuthology.orchestra.run.smithi066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T03:29:05.728 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T03:29:05.728 INFO:teuthology.orchestra.run.smithi066.stdout: debug rgw = 20 2020-11-19T03:29:05.729 INFO:teuthology.orchestra.run.smithi066.stdout: rgw crypt require ssl = False 2020-11-19T03:29:05.729 INFO:teuthology.orchestra.run.smithi066.stdout: rgw crypt s3 kms backend = testing 2020-11-19T03:29:05.729 INFO:teuthology.orchestra.run.smithi066.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T03:29:05.729 INFO:teuthology.orchestra.run.smithi066.stdout: rgw curl low speed time = 300 2020-11-19T03:29:05.729 INFO:teuthology.orchestra.run.smithi066.stdout: rgw data log num shards = 4 2020-11-19T03:29:05.730 INFO:teuthology.orchestra.run.smithi066.stdout: rgw md log max shards = 4 2020-11-19T03:29:05.730 INFO:teuthology.orchestra.run.smithi066.stdout: rgw sync log trim interval = 0 2020-11-19T03:29:05.730 INFO:teuthology.orchestra.run.smithi066.stdout: rgw sync obj etag verify = True 2020-11-19T03:29:05.730 INFO:teuthology.orchestra.run.smithi066.stdout: setgroup = ceph 2020-11-19T03:29:05.730 INFO:teuthology.orchestra.run.smithi066.stdout: setuser = ceph 2020-11-19T03:29:05.730 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.a] 2020-11-19T03:29:05.731 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.0] 2020-11-19T03:29:05.731 INFO:teuthology.orchestra.run.smithi066.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T03:29:05.731 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.1] 2020-11-19T03:29:05.731 INFO:teuthology.orchestra.run.smithi066.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T03:29:05.731 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.2] 2020-11-19T03:29:05.732 INFO:teuthology.orchestra.run.smithi066.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T03:29:05.733 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2020-11-19T03:29:05.733 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2020-11-19T03:29:05.733 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-19T03:29:05.734 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2020-11-19T03:29:05.734 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.734 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2020-11-19T03:29:05.734 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.734 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2020-11-19T03:29:05.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.735 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2020-11-19T03:29:05.735 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2020-11-19T03:29:05.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.735 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2020-11-19T03:29:05.736 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2020-11-19T03:29:05.736 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-19T03:29:05.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2020-11-19T03:29:05.736 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.737 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2020-11-19T03:29:05.737 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-19T03:29:05.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.737 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2020-11-19T03:29:05.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.738 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2020-11-19T03:29:05.738 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2020-11-19T03:29:05.738 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2020-11-19T03:29:05.738 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2020-11-19T03:29:05.738 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2020-11-19T03:29:05.739 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2020-11-19T03:29:05.739 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2020-11-19T03:29:05.739 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-19T03:29:05.739 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-19T03:29:05.739 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2020-11-19T03:29:05.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.740 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-19T03:29:05.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.740 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2020-11-19T03:29:05.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.740 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2020-11-19T03:29:05.741 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.741 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2020-11-19T03:29:05.741 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2020-11-19T03:29:05.741 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2020-11-19T03:29:05.741 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = 172.21.15.154 2020-11-19T03:29:05.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd heartbeat grace = 40 2020-11-19T03:29:05.742 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = d81238a5-f5eb-4f92-b40e-e63250eb12be 2020-11-19T03:29:05.742 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.742 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2020-11-19T03:29:05.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2020-11-19T03:29:05.743 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.743 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2020-11-19T03:29:05.743 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2020-11-19T03:29:05.743 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.743 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2020-11-19T03:29:05.744 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2020-11-19T03:29:05.744 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.744 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2020-11-19T03:29:05.744 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2020-11-19T03:29:05.744 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2020-11-19T03:29:05.745 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.745 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2020-11-19T03:29:05.745 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2020-11-19T03:29:05.745 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.745 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2020-11-19T03:29:05.745 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.746 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2020-11-19T03:29:05.746 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.746 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2020-11-19T03:29:05.746 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.746 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2020-11-19T03:29:05.747 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2020-11-19T03:29:05.747 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.747 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2020-11-19T03:29:05.747 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2020-11-19T03:29:05.747 INFO:teuthology.orchestra.run.smithi154.stdout: debug filestore = 20 2020-11-19T03:29:05.748 INFO:teuthology.orchestra.run.smithi154.stdout: debug journal = 20 2020-11-19T03:29:05.748 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 20 2020-11-19T03:29:05.748 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 25 2020-11-19T03:29:05.748 INFO:teuthology.orchestra.run.smithi154.stdout: osd fast shutdown = False 2020-11-19T03:29:05.748 INFO:teuthology.orchestra.run.smithi154.stdout: rocksdb delete range threshold = 0 2020-11-19T03:29:05.749 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.749 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2020-11-19T03:29:05.749 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2020-11-19T03:29:05.749 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2020-11-19T03:29:05.749 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2020-11-19T03:29:05.750 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2020-11-19T03:29:05.750 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:29:05.750 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:29:05.750 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2020-11-19T03:29:05.750 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.751 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2020-11-19T03:29:05.751 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2020-11-19T03:29:05.751 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2020-11-19T03:29:05.751 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2020-11-19T03:29:05.751 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2020-11-19T03:29:05.752 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2020-11-19T03:29:05.752 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2020-11-19T03:29:05.752 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2020-11-19T03:29:05.752 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2020-11-19T03:29:05.752 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2020-11-19T03:29:05.752 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2020-11-19T03:29:05.753 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd crush smoke test = False 2020-11-19T03:29:05.753 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:05.753 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2020-11-19T03:29:05.753 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2020-11-19T03:29:05.753 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2020-11-19T03:29:05.754 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2020-11-19T03:29:05.754 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-19T03:29:05.754 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-19T03:29:05.754 INFO:teuthology.orchestra.run.smithi154.stdout: debug rgw = 20 2020-11-19T03:29:05.754 INFO:teuthology.orchestra.run.smithi154.stdout: rgw crypt require ssl = False 2020-11-19T03:29:05.755 INFO:teuthology.orchestra.run.smithi154.stdout: rgw crypt s3 kms backend = testing 2020-11-19T03:29:05.755 INFO:teuthology.orchestra.run.smithi154.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-19T03:29:05.755 INFO:teuthology.orchestra.run.smithi154.stdout: rgw curl low speed time = 300 2020-11-19T03:29:05.755 INFO:teuthology.orchestra.run.smithi154.stdout: rgw data log num shards = 4 2020-11-19T03:29:05.755 INFO:teuthology.orchestra.run.smithi154.stdout: rgw md log max shards = 4 2020-11-19T03:29:05.756 INFO:teuthology.orchestra.run.smithi154.stdout: rgw sync log trim interval = 0 2020-11-19T03:29:05.756 INFO:teuthology.orchestra.run.smithi154.stdout: rgw sync obj etag verify = True 2020-11-19T03:29:05.756 INFO:teuthology.orchestra.run.smithi154.stdout: setgroup = ceph 2020-11-19T03:29:05.756 INFO:teuthology.orchestra.run.smithi154.stdout: setuser = ceph 2020-11-19T03:29:05.756 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2020-11-19T03:29:05.756 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2020-11-19T03:29:05.757 INFO:teuthology.orchestra.run.smithi154.stdout: osd_max_omap_entries_per_request = 10 2020-11-19T03:29:05.757 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2020-11-19T03:29:05.757 INFO:teuthology.orchestra.run.smithi154.stdout: osd_max_omap_entries_per_request = 1000 2020-11-19T03:29:05.757 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2020-11-19T03:29:05.757 INFO:teuthology.orchestra.run.smithi154.stdout: osd_max_omap_entries_per_request = 10000 2020-11-19T03:29:05.758 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-11-19T03:29:05.759 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-11-19T03:29:05.846 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-19T03:29:05.846 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:05.847 INFO:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-19T03:29:05.851 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-19T03:29:05.853 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.] 2020-11-19T03:29:05.853 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCB5rVfbYqwGRAAgycGzbdMJa1MsKhbQ3ab1w== 2020-11-19T03:29:05.853 INFO:teuthology.orchestra.run.smithi154.stdout:[client.admin] 2020-11-19T03:29:05.853 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCB5rVfZxM8MhAAZDvdyG6V80p0THqsC4mAJw== 2020-11-19T03:29:05.854 INFO:teuthology.orchestra.run.smithi154.stdout: caps mds = "allow *" 2020-11-19T03:29:05.854 INFO:teuthology.orchestra.run.smithi154.stdout: caps mgr = "allow *" 2020-11-19T03:29:05.854 INFO:teuthology.orchestra.run.smithi154.stdout: caps mon = "allow *" 2020-11-19T03:29:05.854 INFO:teuthology.orchestra.run.smithi154.stdout: caps osd = "allow *" 2020-11-19T03:29:05.855 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:05.855 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:05.855 INFO:teuthology.orchestra.run.smithi154.stderr:206 bytes copied, 0.000147323 s, 1.4 MB/s 2020-11-19T03:29:05.855 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:05.856 INFO:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-19T03:29:05.905 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-19T03:29:05.906 INFO:teuthology.orchestra.run.smithi154.stdout: ��8���O���2P����_18S$��_18S$�aXa �����a 2020-11-19T03:29:05.907 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:05.907 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:05.907 INFO:teuthology.orchestra.run.smithi154.stderr:201 bytes copied, 0.000156594 s, 1.3 MB/s 2020-11-19T03:29:05.908 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi066.front.sepia.ceph.com 2020-11-19T03:29:05.908 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:29:05.909 INFO:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/c2.keyring 2020-11-19T03:29:05.909 INFO:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T03:29:05.914 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-19T03:29:05.924 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:29:05.924 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:29:05.924 INFO:teuthology.orchestra.run.smithi066.stderr:206 bytes copied, 6.4543e-05 s, 3.2 MB/s 2020-11-19T03:29:05.925 INFO:teuthology.orchestra.run.smithi066.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T03:29:05.934 INFO:teuthology.orchestra.run.smithi066:> set -ex 2020-11-19T03:29:05.935 INFO:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:05.986 INFO:teuthology.orchestra.run.smithi066.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:05.987 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records in 2020-11-19T03:29:05.987 INFO:teuthology.orchestra.run.smithi066.stderr:0+1 records out 2020-11-19T03:29:05.988 INFO:teuthology.orchestra.run.smithi066.stderr:201 bytes copied, 7.0894e-05 s, 2.8 MB/s 2020-11-19T03:29:05.988 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:29:05.988 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:05.989 INFO:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/c2.keyring 2020-11-19T03:29:05.989 INFO:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T03:29:05.994 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-19T03:29:06.004 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:06.004 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:06.004 INFO:teuthology.orchestra.run.smithi154.stderr:206 bytes copied, 0.00012201 s, 1.7 MB/s 2020-11-19T03:29:06.005 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-19T03:29:06.014 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:06.014 INFO:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:06.064 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:06.106 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:06.106 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:06.106 INFO:teuthology.orchestra.run.smithi154.stderr:201 bytes copied, 0.0426779 s, 4.7 kB/s 2020-11-19T03:29:06.106 INFO:tasks.ceph:Setting up mon nodes... 2020-11-19T03:29:06.107 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-19T03:29:06.107 INFO:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-11-19T03:29:06.188 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-11-19T03:29:06.191 INFO:tasks.ceph:Setting up mds nodes... 2020-11-19T03:29:06.191 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-19T03:29:06.192 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-11-19T03:29:06.265 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/c2.client.0.keyring 2020-11-19T03:29:06.277 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-11-19T03:29:06.354 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/c2.client.1.keyring 2020-11-19T03:29:06.365 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-19T03:29:06.366 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-19T03:29:06.366 INFO:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-11-19T03:29:06.424 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:29:06.425 INFO:tasks.ceph:role: c2.osd.0 2020-11-19T03:29:06.425 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:29:06.426 INFO:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-19T03:29:06.951 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T03:29:06.952 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T03:29:06.952 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T03:29:06.954 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T03:29:06.954 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2020-11-19T03:29:06.954 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T03:29:06.954 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T03:29:06.955 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T03:29:06.955 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T03:29:06.956 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2020-11-19T03:29:06.956 INFO:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-11-19T03:29:07.032 INFO:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-11-19T03:29:07.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T03:29:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T03:29:07.047 INFO:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-11-19T03:29:07.107 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:29:07.110 INFO:tasks.ceph:role: c2.osd.1 2020-11-19T03:29:07.110 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:29:07.110 INFO:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-19T03:29:07.618 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T03:29:07.618 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T03:29:07.619 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T03:29:07.619 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T03:29:07.619 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2020-11-19T03:29:07.619 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T03:29:07.619 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T03:29:07.620 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T03:29:07.620 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T03:29:07.621 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2020-11-19T03:29:07.621 INFO:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-11-19T03:29:07.687 INFO:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-11-19T03:29:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T03:29:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T03:29:07.719 INFO:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-11-19T03:29:07.790 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-19T03:29:07.790 INFO:tasks.ceph:role: c2.osd.2 2020-11-19T03:29:07.790 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2020-11-19T03:29:07.791 INFO:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-19T03:29:08.314 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-19T03:29:08.314 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-19T03:29:08.315 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-19T03:29:08.315 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-19T03:29:08.315 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2020-11-19T03:29:08.315 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-19T03:29:08.315 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-19T03:29:08.316 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-19T03:29:08.316 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-19T03:29:08.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2020-11-19T03:29:08.319 INFO:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-11-19T03:29:08.370 INFO:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-11-19T03:29:08.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2020-11-19T03:29:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-19T03:29:08.441 INFO:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:08.547+0000 7f2f38f12d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-11-19T03:29:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:08.547+0000 7f2f38f12d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-11-19T03:29:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:08.547+0000 7f2f38f12d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-19T03:29:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:08.547+0000 7f2f38f12d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-19T03:29:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:08.547+0000 7f2f38f12d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-19T03:29:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:08.547+0000 7f2f38f12d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-11-19T03:29:10.724 INFO:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-11-19T03:29:10.738 INFO:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:10.819 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:10.815+0000 7f1552602d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-11-19T03:29:10.819 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:10.815+0000 7f1552602d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-11-19T03:29:10.820 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:10.815+0000 7f1552602d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-19T03:29:10.820 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:10.815+0000 7f1552602d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-19T03:29:10.820 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:10.815+0000 7f1552602d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-19T03:29:10.821 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:10.815+0000 7f1552602d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-11-19T03:29:13.000 INFO:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-11-19T03:29:13.063 INFO:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:13.175 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:13.171+0000 7fa9078f2d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-11-19T03:29:13.175 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:13.171+0000 7fa9078f2d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-11-19T03:29:13.176 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:13.171+0000 7fa9078f2d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-19T03:29:13.176 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:13.171+0000 7fa9078f2d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-19T03:29:13.176 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:13.171+0000 7fa9078f2d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-19T03:29:13.176 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:29:13.171+0000 7fa9078f2d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-11-19T03:29:15.348 INFO:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-11-19T03:29:15.404 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-19T03:29:15.405 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:15.405 INFO:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-19T03:29:15.449 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-19T03:29:15.457 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2020-11-19T03:29:15.458 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCC5rVfP98wCxAAPt66hrWDqrU/duJyfzlMoA== 2020-11-19T03:29:15.458 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:15.459 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:15.459 INFO:teuthology.orchestra.run.smithi154.stderr:56 bytes copied, 0.000205564 s, 272 kB/s 2020-11-19T03:29:15.460 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:15.460 INFO:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-19T03:29:15.504 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-19T03:29:15.515 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2020-11-19T03:29:15.516 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCE5rVfs7usIBAA0Uk0z/xg3qOcYyOVwCsCGA== 2020-11-19T03:29:15.516 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:15.516 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:15.516 INFO:teuthology.orchestra.run.smithi154.stderr:56 bytes copied, 0.000161537 s, 347 kB/s 2020-11-19T03:29:15.517 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:15.518 INFO:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-19T03:29:15.560 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-19T03:29:15.570 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2020-11-19T03:29:15.570 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCG5rVfzry7MBAAJ7RRGsmMcKkEUbeTOb3RTA== 2020-11-19T03:29:15.570 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:15.571 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:15.571 INFO:teuthology.orchestra.run.smithi154.stderr:56 bytes copied, 0.000160051 s, 350 kB/s 2020-11-19T03:29:15.572 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:15.572 INFO:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-19T03:29:15.616 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-19T03:29:15.624 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2020-11-19T03:29:15.625 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCJ5rVfpyxfChAAujm+nzwk1Thq3M6TqK7Sqg== 2020-11-19T03:29:15.625 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:15.625 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:15.625 INFO:teuthology.orchestra.run.smithi154.stderr:56 bytes copied, 0.000197461 s, 284 kB/s 2020-11-19T03:29:15.628 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:15.628 INFO:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-19T03:29:15.672 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-19T03:29:15.674 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2020-11-19T03:29:15.674 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCC5rVf/q7BDxAAtUyoYeO8CNJ2zSUa4XrorQ== 2020-11-19T03:29:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:59 bytes copied, 0.000166037 s, 355 kB/s 2020-11-19T03:29:15.675 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:15.676 INFO:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-19T03:29:15.721 INFO:teuthology.orchestra.run.smithi154.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-19T03:29:15.722 INFO:teuthology.orchestra.run.smithi154.stdout:[client.1] 2020-11-19T03:29:15.723 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCC5rVfMewGFRAA/3KrN3kRwO6M9dqFoofcGQ== 2020-11-19T03:29:15.723 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:15.723 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:15.724 INFO:teuthology.orchestra.run.smithi154.stderr:59 bytes copied, 0.000161984 s, 364 kB/s 2020-11-19T03:29:15.724 INFO:tasks.ceph:Adding keys to all mons... 2020-11-19T03:29:15.724 INFO:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/c2.keyring 2020-11-19T03:29:15.777 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2020-11-19T03:29:15.778 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCC5rVfP98wCxAAPt66hrWDqrU/duJyfzlMoA== 2020-11-19T03:29:15.778 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2020-11-19T03:29:15.778 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCE5rVfs7usIBAA0Uk0z/xg3qOcYyOVwCsCGA== 2020-11-19T03:29:15.778 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2020-11-19T03:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCG5rVfzry7MBAAJ7RRGsmMcKkEUbeTOb3RTA== 2020-11-19T03:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2020-11-19T03:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCJ5rVfpyxfChAAujm+nzwk1Thq3M6TqK7Sqg== 2020-11-19T03:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2020-11-19T03:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCC5rVf/q7BDxAAtUyoYeO8CNJ2zSUa4XrorQ== 2020-11-19T03:29:15.780 INFO:teuthology.orchestra.run.smithi154.stdout:[client.1] 2020-11-19T03:29:15.780 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCC5rVfMewGFRAA/3KrN3kRwO6M9dqFoofcGQ== 2020-11-19T03:29:15.781 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-19T03:29:15.866 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T03:29:15.950 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T03:29:16.037 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-19T03:29:16.074 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T03:29:16.161 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-19T03:29:16.250 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-19T03:29:16.251 INFO:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-11-19T03:29:16.266 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-11-19T03:29:16.363 INFO:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-11-19T03:29:16.389 INFO:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-11-19T03:29:16.452 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-11-19T03:29:16.452 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-11-19T03:29:16.453 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-11-19T03:29:16.509 INFO:tasks.ceph.c2.mon.a:Started 2020-11-19T03:29:16.510 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-11-19T03:29:16.510 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-11-19T03:29:16.511 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-11-19T03:29:16.524 INFO:tasks.ceph.c2.mgr.x:Started 2020-11-19T03:29:16.526 INFO:tasks.ceph:Setting crush tunables to default 2020-11-19T03:29:16.527 INFO:teuthology.orchestra.run.smithi154:> sudo ceph --cluster c2 osd crush tunables default 2020-11-19T03:29:16.982 INFO:teuthology.orchestra.run.smithi154.stderr:adjusted tunables profile to default 2020-11-19T03:29:16.993 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-11-19T03:29:16.994 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:16.994 INFO:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-19T03:29:17.020 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-19T03:29:17.029 INFO:teuthology.orchestra.run.smithi154.stdout:7af457d1-e720-42d9-8e93-c8e1cc1f7965 2020-11-19T03:29:17.030 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:17.030 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:17.030 INFO:teuthology.orchestra.run.smithi154.stderr:37 bytes copied, 0.000163191 s, 227 kB/s 2020-11-19T03:29:17.031 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:17.031 INFO:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-19T03:29:17.090 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-19T03:29:17.098 INFO:teuthology.orchestra.run.smithi154.stdout:69676815-faaf-43bb-a419-5fd481cd7973 2020-11-19T03:29:17.098 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:17.098 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:17.099 INFO:teuthology.orchestra.run.smithi154.stderr:37 bytes copied, 0.000171554 s, 216 kB/s 2020-11-19T03:29:17.100 INFO:teuthology.orchestra.run.smithi154:> set -ex 2020-11-19T03:29:17.100 INFO:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-19T03:29:17.159 INFO:teuthology.orchestra.run.smithi154.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-19T03:29:17.167 INFO:teuthology.orchestra.run.smithi154.stdout:9a0cb440-bdea-4fd7-83e0-fe4a45be3719 2020-11-19T03:29:17.168 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2020-11-19T03:29:17.168 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2020-11-19T03:29:17.169 INFO:teuthology.orchestra.run.smithi154.stderr:37 bytes copied, 9.1407e-05 s, 405 kB/s 2020-11-19T03:29:17.169 INFO:teuthology.orchestra.run.smithi154:> sudo ceph --cluster c2 osd new 7af457d1-e720-42d9-8e93-c8e1cc1f7965 0 2020-11-19T03:29:17.432 INFO:teuthology.orchestra.run.smithi154.stdout:0 2020-11-19T03:29:17.447 INFO:teuthology.orchestra.run.smithi154:> sudo ceph --cluster c2 osd new 69676815-faaf-43bb-a419-5fd481cd7973 1 2020-11-19T03:29:17.660 INFO:teuthology.orchestra.run.smithi154.stdout:1 2020-11-19T03:29:17.677 INFO:teuthology.orchestra.run.smithi154:> sudo ceph --cluster c2 osd new 9a0cb440-bdea-4fd7-83e0-fe4a45be3719 2 2020-11-19T03:29:17.913 INFO:teuthology.orchestra.run.smithi154.stdout:2 2020-11-19T03:29:17.928 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-11-19T03:29:17.929 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-11-19T03:29:17.948 INFO:tasks.ceph.c2.osd.0:Started 2020-11-19T03:29:17.948 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-11-19T03:29:17.949 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-11-19T03:29:17.965 INFO:tasks.ceph.c2.osd.1:Started 2020-11-19T03:29:17.970 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-11-19T03:29:17.970 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-11-19T03:29:17.986 INFO:tasks.ceph.c2.osd.2:Started 2020-11-19T03:29:17.986 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-19T03:29:18.289 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-19T03:29:18.323 INFO:tasks.ceph.c2.osd.1.smithi154.stderr:2020-11-19T03:29:18.319+0000 7efc1c0f7d80 -1 Falling back to public interface 2020-11-19T03:29:18.358 INFO:tasks.ceph.c2.osd.2.smithi154.stderr:2020-11-19T03:29:18.355+0000 7f575a116d80 -1 Falling back to public interface 2020-11-19T03:29:18.480 INFO:teuthology.misc.health.smithi154.stdout: 2020-11-19T03:29:18.481 INFO:teuthology.misc.health.smithi154.stdout:{"epoch":5,"fsid":"d81238a5-f5eb-4f92-b40e-e63250eb12be","created":"2020-11-19T03:29:16.622630+0000","modified":"2020-11-19T03:29:17.909574+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7af457d1-e720-42d9-8e93-c8e1cc1f7965","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69676815-faaf-43bb-a419-5fd481cd7973","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9a0cb440-bdea-4fd7-83e0-fe4a45be3719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:29:18.492 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-19T03:29:18.562 INFO:tasks.ceph.c2.osd.0.smithi154.stderr:2020-11-19T03:29:18.559+0000 7f882d66fd80 -1 Falling back to public interface 2020-11-19T03:29:18.945 INFO:tasks.ceph.c2.osd.1.smithi154.stderr:2020-11-19T03:29:18.943+0000 7efc1c0f7d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-19T03:29:19.012 INFO:tasks.ceph.c2.osd.2.smithi154.stderr:2020-11-19T03:29:19.007+0000 7f575a116d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-19T03:29:19.202 INFO:tasks.ceph.c2.osd.0.smithi154.stderr:2020-11-19T03:29:19.199+0000 7f882d66fd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-19T03:29:20.652 INFO:tasks.ceph.c2.osd.1.smithi154.stderr:2020-11-19T03:29:20.647+0000 7efc135c1700 -1 osd.1 0 waiting for initial osdmap 2020-11-19T03:29:20.653 INFO:tasks.ceph.c2.osd.0.smithi154.stderr:2020-11-19T03:29:20.647+0000 7f8824b39700 -1 osd.0 0 waiting for initial osdmap 2020-11-19T03:29:20.653 INFO:tasks.ceph.c2.osd.2.smithi154.stderr:2020-11-19T03:29:20.651+0000 7f5752de3700 -1 osd.2 0 waiting for initial osdmap 2020-11-19T03:29:24.797 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-19T03:29:25.084 INFO:teuthology.misc.health.smithi154.stdout: 2020-11-19T03:29:25.084 INFO:teuthology.misc.health.smithi154.stdout:{"epoch":10,"fsid":"d81238a5-f5eb-4f92-b40e-e63250eb12be","created":"2020-11-19T03:29:16.622630+0000","modified":"2020-11-19T03:29:23.668451+0000","last_up_change":"2020-11-19T03:29:21.648104+0000","last_in_change":"2020-11-19T03:29:21.648104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:29:20.532431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7af457d1-e720-42d9-8e93-c8e1cc1f7965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":11311},{"type":"v1","addr":"172.21.15.154:6817","nonce":11311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":11311},{"type":"v1","addr":"172.21.15.154:6819","nonce":11311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":11311},{"type":"v1","addr":"172.21.15.154:6823","nonce":11311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":11311},{"type":"v1","addr":"172.21.15.154:6821","nonce":11311}]},"public_addr":"172.21.15.154:6817/11311","cluster_addr":"172.21.15.154:6819/11311","heartbeat_back_addr":"172.21.15.154:6823/11311","heartbeat_front_addr":"172.21.15.154:6821/11311","state":["exists","up"]},{"osd":1,"uuid":"69676815-faaf-43bb-a419-5fd481cd7973","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":11320},{"type":"v1","addr":"172.21.15.154:6801","nonce":11320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":11320},{"type":"v1","addr":"172.21.15.154:6803","nonce":11320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":11320},{"type":"v1","addr":"172.21.15.154:6807","nonce":11320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":11320},{"type":"v1","addr":"172.21.15.154:6805","nonce":11320}]},"public_addr":"172.21.15.154:6801/11320","cluster_addr":"172.21.15.154:6803/11320","heartbeat_back_addr":"172.21.15.154:6807/11320","heartbeat_front_addr":"172.21.15.154:6805/11320","state":["exists","up"]},{"osd":2,"uuid":"9a0cb440-bdea-4fd7-83e0-fe4a45be3719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":11341},{"type":"v1","addr":"172.21.15.154:6809","nonce":11341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":11341},{"type":"v1","addr":"172.21.15.154:6811","nonce":11341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":11341},{"type":"v1","addr":"172.21.15.154:6815","nonce":11341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":11341},{"type":"v1","addr":"172.21.15.154:6813","nonce":11341}]},"public_addr":"172.21.15.154:6809/11341","cluster_addr":"172.21.15.154:6811/11341","heartbeat_back_addr":"172.21.15.154:6815/11341","heartbeat_front_addr":"172.21.15.154:6813/11341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.241790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:19.911900+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.047651+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:29:25.097 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-19T03:29:25.097 INFO:tasks.ceph:Creating RBD pool 2020-11-19T03:29:25.098 INFO:teuthology.orchestra.run.smithi154:> sudo ceph --cluster c2 osd pool create rbd 8 2020-11-19T03:29:25.704 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'rbd' created 2020-11-19T03:29:25.718 INFO:teuthology.orchestra.run.smithi154:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-19T03:29:26.713 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-19T03:29:26.727 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-11-19T03:29:26.728 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-19T03:29:26.728 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-19T03:29:27.019 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:27.020 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":12,"fsid":"d81238a5-f5eb-4f92-b40e-e63250eb12be","created":"2020-11-19T03:29:16.622630+0000","modified":"2020-11-19T03:29:26.689210+0000","last_up_change":"2020-11-19T03:29:21.648104+0000","last_in_change":"2020-11-19T03:29:21.648104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:29:20.532431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T03:29:25.388590+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7af457d1-e720-42d9-8e93-c8e1cc1f7965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":11311},{"type":"v1","addr":"172.21.15.154:6817","nonce":11311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":11311},{"type":"v1","addr":"172.21.15.154:6819","nonce":11311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":11311},{"type":"v1","addr":"172.21.15.154:6823","nonce":11311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":11311},{"type":"v1","addr":"172.21.15.154:6821","nonce":11311}]},"public_addr":"172.21.15.154:6817/11311","cluster_addr":"172.21.15.154:6819/11311","heartbeat_back_addr":"172.21.15.154:6823/11311","heartbeat_front_addr":"172.21.15.154:6821/11311","state":["exists","up"]},{"osd":1,"uuid":"69676815-faaf-43bb-a419-5fd481cd7973","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":11320},{"type":"v1","addr":"172.21.15.154:6801","nonce":11320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":11320},{"type":"v1","addr":"172.21.15.154:6803","nonce":11320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":11320},{"type":"v1","addr":"172.21.15.154:6807","nonce":11320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":11320},{"type":"v1","addr":"172.21.15.154:6805","nonce":11320}]},"public_addr":"172.21.15.154:6801/11320","cluster_addr":"172.21.15.154:6803/11320","heartbeat_back_addr":"172.21.15.154:6807/11320","heartbeat_front_addr":"172.21.15.154:6805/11320","state":["exists","up"]},{"osd":2,"uuid":"9a0cb440-bdea-4fd7-83e0-fe4a45be3719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":11341},{"type":"v1","addr":"172.21.15.154:6809","nonce":11341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":11341},{"type":"v1","addr":"172.21.15.154:6811","nonce":11341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":11341},{"type":"v1","addr":"172.21.15.154:6815","nonce":11341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":11341},{"type":"v1","addr":"172.21.15.154:6813","nonce":11341}]},"public_addr":"172.21.15.154:6809/11341","cluster_addr":"172.21.15.154:6811/11341","heartbeat_back_addr":"172.21.15.154:6815/11341","heartbeat_front_addr":"172.21.15.154:6813/11341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.241790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:19.911900+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.047651+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:29:27.034 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-19T03:29:20.532431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-19T03:29:25.388590+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-19T03:29:27.035 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2020-11-19T03:29:27.332 INFO:teuthology.orchestra.run.smithi154.stdout:pg_num: 1 2020-11-19T03:29:27.346 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2020-11-19T03:29:27.643 INFO:teuthology.orchestra.run.smithi154.stdout:pg_num: 8 2020-11-19T03:29:27.658 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-11-19T03:29:27.658 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-11-19T03:29:27.658 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2020-11-19T03:29:27.982 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:27.995 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":11128},{"type":"v1","addr":"172.21.15.154:6825","nonce":11128}]},"active_addr":"172.21.15.154:6825/11128","active_change":"2020-11-19T03:29:20.505503+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":754616747}]},{"addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":503814616}]},{"addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":4222159265}]}]}} 2020-11-19T03:29:27.998 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-11-19T03:29:27.998 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-11-19T03:29:27.998 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-19T03:29:28.277 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:28.278 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":13,"fsid":"d81238a5-f5eb-4f92-b40e-e63250eb12be","created":"2020-11-19T03:29:16.622630+0000","modified":"2020-11-19T03:29:27.695991+0000","last_up_change":"2020-11-19T03:29:21.648104+0000","last_in_change":"2020-11-19T03:29:21.648104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:29:20.532431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T03:29:25.388590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7af457d1-e720-42d9-8e93-c8e1cc1f7965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":11311},{"type":"v1","addr":"172.21.15.154:6817","nonce":11311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":11311},{"type":"v1","addr":"172.21.15.154:6819","nonce":11311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":11311},{"type":"v1","addr":"172.21.15.154:6823","nonce":11311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":11311},{"type":"v1","addr":"172.21.15.154:6821","nonce":11311}]},"public_addr":"172.21.15.154:6817/11311","cluster_addr":"172.21.15.154:6819/11311","heartbeat_back_addr":"172.21.15.154:6823/11311","heartbeat_front_addr":"172.21.15.154:6821/11311","state":["exists","up"]},{"osd":1,"uuid":"69676815-faaf-43bb-a419-5fd481cd7973","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":11320},{"type":"v1","addr":"172.21.15.154:6801","nonce":11320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":11320},{"type":"v1","addr":"172.21.15.154:6803","nonce":11320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":11320},{"type":"v1","addr":"172.21.15.154:6807","nonce":11320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":11320},{"type":"v1","addr":"172.21.15.154:6805","nonce":11320}]},"public_addr":"172.21.15.154:6801/11320","cluster_addr":"172.21.15.154:6803/11320","heartbeat_back_addr":"172.21.15.154:6807/11320","heartbeat_front_addr":"172.21.15.154:6805/11320","state":["exists","up"]},{"osd":2,"uuid":"9a0cb440-bdea-4fd7-83e0-fe4a45be3719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":11341},{"type":"v1","addr":"172.21.15.154:6809","nonce":11341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":11341},{"type":"v1","addr":"172.21.15.154:6811","nonce":11341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":11341},{"type":"v1","addr":"172.21.15.154:6815","nonce":11341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":11341},{"type":"v1","addr":"172.21.15.154:6813","nonce":11341}]},"public_addr":"172.21.15.154:6809/11341","cluster_addr":"172.21.15.154:6811/11341","heartbeat_back_addr":"172.21.15.154:6815/11341","heartbeat_front_addr":"172.21.15.154:6813/11341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.241790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:19.911900+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.047651+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:29:28.291 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-11-19T03:29:28.292 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-19T03:29:28.583 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:28.584 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":13,"fsid":"d81238a5-f5eb-4f92-b40e-e63250eb12be","created":"2020-11-19T03:29:16.622630+0000","modified":"2020-11-19T03:29:27.695991+0000","last_up_change":"2020-11-19T03:29:21.648104+0000","last_in_change":"2020-11-19T03:29:21.648104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-19T03:29:20.532431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-19T03:29:25.388590+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7af457d1-e720-42d9-8e93-c8e1cc1f7965","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":11311},{"type":"v1","addr":"172.21.15.154:6817","nonce":11311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":11311},{"type":"v1","addr":"172.21.15.154:6819","nonce":11311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":11311},{"type":"v1","addr":"172.21.15.154:6823","nonce":11311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":11311},{"type":"v1","addr":"172.21.15.154:6821","nonce":11311}]},"public_addr":"172.21.15.154:6817/11311","cluster_addr":"172.21.15.154:6819/11311","heartbeat_back_addr":"172.21.15.154:6823/11311","heartbeat_front_addr":"172.21.15.154:6821/11311","state":["exists","up"]},{"osd":1,"uuid":"69676815-faaf-43bb-a419-5fd481cd7973","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":11320},{"type":"v1","addr":"172.21.15.154:6801","nonce":11320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":11320},{"type":"v1","addr":"172.21.15.154:6803","nonce":11320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":11320},{"type":"v1","addr":"172.21.15.154:6807","nonce":11320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":11320},{"type":"v1","addr":"172.21.15.154:6805","nonce":11320}]},"public_addr":"172.21.15.154:6801/11320","cluster_addr":"172.21.15.154:6803/11320","heartbeat_back_addr":"172.21.15.154:6807/11320","heartbeat_front_addr":"172.21.15.154:6805/11320","state":["exists","up"]},{"osd":2,"uuid":"9a0cb440-bdea-4fd7-83e0-fe4a45be3719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":11341},{"type":"v1","addr":"172.21.15.154:6809","nonce":11341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":11341},{"type":"v1","addr":"172.21.15.154:6811","nonce":11341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":11341},{"type":"v1","addr":"172.21.15.154:6815","nonce":11341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":11341},{"type":"v1","addr":"172.21.15.154:6813","nonce":11341}]},"public_addr":"172.21.15.154:6809/11341","cluster_addr":"172.21.15.154:6811/11341","heartbeat_back_addr":"172.21.15.154:6815/11341","heartbeat_front_addr":"172.21.15.154:6813/11341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.241790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:19.911900+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-19T03:29:20.047651+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-19T03:29:28.598 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2020-11-19T03:29:28.790 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2020-11-19T03:29:28.791 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2020-11-19T03:29:28.968 INFO:teuthology.orchestra.run.smithi154.stdout:34359738371 2020-11-19T03:29:30.512 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2020-11-19T03:29:30.691 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2020-11-19T03:29:30.692 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-19T03:29:30.989 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2020-11-19T03:29:31.003 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2020-11-19T03:29:31.004 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2020-11-19T03:29:31.284 INFO:teuthology.orchestra.run.smithi154.stdout:34359738371 2020-11-19T03:29:31.298 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-11-19T03:29:31.299 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2020-11-19T03:29:31.583 INFO:teuthology.orchestra.run.smithi154.stdout:34359738371 2020-11-19T03:29:31.603 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.2 2020-11-19T03:29:32.604 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2020-11-19T03:29:32.911 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2020-11-19T03:29:32.930 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-11-19T03:29:32.930 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-11-19T03:29:32.931 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-19T03:29:33.216 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:33.218 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2020-11-19T03:29:33.230 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-19T03:29:32.515995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14760,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558040,"statfs":{"total":322122547200,"available":322107432960,"internally_reserved":0,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817663"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714329+0000","last_change":"2020-11-19T03:29:26.712525+0000","last_active":"2020-11-19T03:29:26.714329+0000","last_peered":"2020-11-19T03:29:26.714329+0000","last_clean":"2020-11-19T03:29:26.714329+0000","last_became_active":"2020-11-19T03:29:26.712012+0000","last_became_peered":"2020-11-19T03:29:26.712012+0000","last_unstale":"2020-11-19T03:29:26.714329+0000","last_undegraded":"2020-11-19T03:29:26.714329+0000","last_fullsized":"2020-11-19T03:29:26.714329+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714303+0000","last_change":"2020-11-19T03:29:26.711170+0000","last_active":"2020-11-19T03:29:26.714303+0000","last_peered":"2020-11-19T03:29:26.714303+0000","last_clean":"2020-11-19T03:29:26.714303+0000","last_became_active":"2020-11-19T03:29:26.710732+0000","last_became_peered":"2020-11-19T03:29:26.710732+0000","last_unstale":"2020-11-19T03:29:26.714303+0000","last_undegraded":"2020-11-19T03:29:26.714303+0000","last_fullsized":"2020-11-19T03:29:26.714303+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714270+0000","last_change":"2020-11-19T03:29:26.711323+0000","last_active":"2020-11-19T03:29:26.714270+0000","last_peered":"2020-11-19T03:29:26.714270+0000","last_clean":"2020-11-19T03:29:26.714270+0000","last_became_active":"2020-11-19T03:29:26.710912+0000","last_became_peered":"2020-11-19T03:29:26.710912+0000","last_unstale":"2020-11-19T03:29:26.714270+0000","last_undegraded":"2020-11-19T03:29:26.714270+0000","last_fullsized":"2020-11-19T03:29:26.714270+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.710717+0000","last_change":"2020-11-19T03:29:26.708605+0000","last_active":"2020-11-19T03:29:26.710717+0000","last_peered":"2020-11-19T03:29:26.710717+0000","last_clean":"2020-11-19T03:29:26.710717+0000","last_became_active":"2020-11-19T03:29:26.708297+0000","last_became_peered":"2020-11-19T03:29:26.708297+0000","last_unstale":"2020-11-19T03:29:26.710717+0000","last_undegraded":"2020-11-19T03:29:26.710717+0000","last_fullsized":"2020-11-19T03:29:26.710717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.706052+0000","last_change":"2020-11-19T03:29:23.335535+0000","last_active":"2020-11-19T03:29:26.706052+0000","last_peered":"2020-11-19T03:29:26.706052+0000","last_clean":"2020-11-19T03:29:26.706052+0000","last_became_active":"2020-11-19T03:29:23.333813+0000","last_became_peered":"2020-11-19T03:29:23.333813+0000","last_unstale":"2020-11-19T03:29:26.706052+0000","last_undegraded":"2020-11-19T03:29:26.706052+0000","last_fullsized":"2020-11-19T03:29:26.706052+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:20.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:20.640442+0000","last_clean_scrub_stamp":"2020-11-19T03:29:20.640442+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714300+0000","last_change":"2020-11-19T03:29:26.712806+0000","last_active":"2020-11-19T03:29:26.714300+0000","last_peered":"2020-11-19T03:29:26.714300+0000","last_clean":"2020-11-19T03:29:26.714300+0000","last_became_active":"2020-11-19T03:29:26.712372+0000","last_became_peered":"2020-11-19T03:29:26.712372+0000","last_unstale":"2020-11-19T03:29:26.714300+0000","last_undegraded":"2020-11-19T03:29:26.714300+0000","last_fullsized":"2020-11-19T03:29:26.714300+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.710630+0000","last_change":"2020-11-19T03:29:26.709592+0000","last_active":"2020-11-19T03:29:26.710630+0000","last_peered":"2020-11-19T03:29:26.710630+0000","last_clean":"2020-11-19T03:29:26.710630+0000","last_became_active":"2020-11-19T03:29:26.709067+0000","last_became_peered":"2020-11-19T03:29:26.709067+0000","last_unstale":"2020-11-19T03:29:26.710630+0000","last_undegraded":"2020-11-19T03:29:26.710630+0000","last_fullsized":"2020-11-19T03:29:26.710630+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.716428+0000","last_change":"2020-11-19T03:29:26.713016+0000","last_active":"2020-11-19T03:29:26.716428+0000","last_peered":"2020-11-19T03:29:26.716428+0000","last_clean":"2020-11-19T03:29:26.716428+0000","last_became_active":"2020-11-19T03:29:26.712624+0000","last_became_peered":"2020-11-19T03:29:26.712624+0000","last_unstale":"2020-11-19T03:29:26.716428+0000","last_undegraded":"2020-11-19T03:29:26.716428+0000","last_fullsized":"2020-11-19T03:29:26.716428+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.713609+0000","last_change":"2020-11-19T03:29:26.710189+0000","last_active":"2020-11-19T03:29:26.713609+0000","last_peered":"2020-11-19T03:29:26.713609+0000","last_clean":"2020-11-19T03:29:26.713609+0000","last_became_active":"2020-11-19T03:29:26.709820+0000","last_became_peered":"2020-11-19T03:29:26.709820+0000","last_unstale":"2020-11-19T03:29:26.713609+0000","last_undegraded":"2020-11-19T03:29:26.713609+0000","last_fullsized":"2020-11-19T03:29:26.713609+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]}]}} 2020-11-19T03:29:33.231 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-19T03:29:33.517 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:33.517 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2020-11-19T03:29:33.524 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-19T03:29:32.515995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14760,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558040,"statfs":{"total":322122547200,"available":322107432960,"internally_reserved":0,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817663"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714329+0000","last_change":"2020-11-19T03:29:26.712525+0000","last_active":"2020-11-19T03:29:26.714329+0000","last_peered":"2020-11-19T03:29:26.714329+0000","last_clean":"2020-11-19T03:29:26.714329+0000","last_became_active":"2020-11-19T03:29:26.712012+0000","last_became_peered":"2020-11-19T03:29:26.712012+0000","last_unstale":"2020-11-19T03:29:26.714329+0000","last_undegraded":"2020-11-19T03:29:26.714329+0000","last_fullsized":"2020-11-19T03:29:26.714329+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714303+0000","last_change":"2020-11-19T03:29:26.711170+0000","last_active":"2020-11-19T03:29:26.714303+0000","last_peered":"2020-11-19T03:29:26.714303+0000","last_clean":"2020-11-19T03:29:26.714303+0000","last_became_active":"2020-11-19T03:29:26.710732+0000","last_became_peered":"2020-11-19T03:29:26.710732+0000","last_unstale":"2020-11-19T03:29:26.714303+0000","last_undegraded":"2020-11-19T03:29:26.714303+0000","last_fullsized":"2020-11-19T03:29:26.714303+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714270+0000","last_change":"2020-11-19T03:29:26.711323+0000","last_active":"2020-11-19T03:29:26.714270+0000","last_peered":"2020-11-19T03:29:26.714270+0000","last_clean":"2020-11-19T03:29:26.714270+0000","last_became_active":"2020-11-19T03:29:26.710912+0000","last_became_peered":"2020-11-19T03:29:26.710912+0000","last_unstale":"2020-11-19T03:29:26.714270+0000","last_undegraded":"2020-11-19T03:29:26.714270+0000","last_fullsized":"2020-11-19T03:29:26.714270+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.710717+0000","last_change":"2020-11-19T03:29:26.708605+0000","last_active":"2020-11-19T03:29:26.710717+0000","last_peered":"2020-11-19T03:29:26.710717+0000","last_clean":"2020-11-19T03:29:26.710717+0000","last_became_active":"2020-11-19T03:29:26.708297+0000","last_became_peered":"2020-11-19T03:29:26.708297+0000","last_unstale":"2020-11-19T03:29:26.710717+0000","last_undegraded":"2020-11-19T03:29:26.710717+0000","last_fullsized":"2020-11-19T03:29:26.710717+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.706052+0000","last_change":"2020-11-19T03:29:23.335535+0000","last_active":"2020-11-19T03:29:26.706052+0000","last_peered":"2020-11-19T03:29:26.706052+0000","last_clean":"2020-11-19T03:29:26.706052+0000","last_became_active":"2020-11-19T03:29:23.333813+0000","last_became_peered":"2020-11-19T03:29:23.333813+0000","last_unstale":"2020-11-19T03:29:26.706052+0000","last_undegraded":"2020-11-19T03:29:26.706052+0000","last_fullsized":"2020-11-19T03:29:26.706052+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:20.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:20.640442+0000","last_clean_scrub_stamp":"2020-11-19T03:29:20.640442+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.714300+0000","last_change":"2020-11-19T03:29:26.712806+0000","last_active":"2020-11-19T03:29:26.714300+0000","last_peered":"2020-11-19T03:29:26.714300+0000","last_clean":"2020-11-19T03:29:26.714300+0000","last_became_active":"2020-11-19T03:29:26.712372+0000","last_became_peered":"2020-11-19T03:29:26.712372+0000","last_unstale":"2020-11-19T03:29:26.714300+0000","last_undegraded":"2020-11-19T03:29:26.714300+0000","last_fullsized":"2020-11-19T03:29:26.714300+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.710630+0000","last_change":"2020-11-19T03:29:26.709592+0000","last_active":"2020-11-19T03:29:26.710630+0000","last_peered":"2020-11-19T03:29:26.710630+0000","last_clean":"2020-11-19T03:29:26.710630+0000","last_became_active":"2020-11-19T03:29:26.709067+0000","last_became_peered":"2020-11-19T03:29:26.709067+0000","last_unstale":"2020-11-19T03:29:26.710630+0000","last_undegraded":"2020-11-19T03:29:26.710630+0000","last_fullsized":"2020-11-19T03:29:26.710630+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.716428+0000","last_change":"2020-11-19T03:29:26.713016+0000","last_active":"2020-11-19T03:29:26.716428+0000","last_peered":"2020-11-19T03:29:26.716428+0000","last_clean":"2020-11-19T03:29:26.716428+0000","last_became_active":"2020-11-19T03:29:26.712624+0000","last_became_peered":"2020-11-19T03:29:26.712624+0000","last_unstale":"2020-11-19T03:29:26.716428+0000","last_undegraded":"2020-11-19T03:29:26.716428+0000","last_fullsized":"2020-11-19T03:29:26.716428+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-19T03:29:26.713609+0000","last_change":"2020-11-19T03:29:26.710189+0000","last_active":"2020-11-19T03:29:26.713609+0000","last_peered":"2020-11-19T03:29:26.713609+0000","last_clean":"2020-11-19T03:29:26.713609+0000","last_became_active":"2020-11-19T03:29:26.709820+0000","last_became_peered":"2020-11-19T03:29:26.709820+0000","last_unstale":"2020-11-19T03:29:26.713609+0000","last_undegraded":"2020-11-19T03:29:26.713609+0000","last_fullsized":"2020-11-19T03:29:26.713609+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-19T03:29:25.683034+0000","last_clean_scrub_stamp":"2020-11-19T03:29:25.683034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]}]}} 2020-11-19T03:29:33.525 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-11-19T03:29:33.526 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-11-19T03:29:33.526 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-11-19T03:29:33.526 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2020-11-19T03:29:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: 2020-11-19T03:29:35.978 INFO:teuthology.orchestra.run.smithi154.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-19T03:29:35.978 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-11-19T03:29:35.979 INFO:teuthology.run_tasks:Running task rgw... 2020-11-19T03:29:35.992 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-11-19T03:29:35.993 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-11-19T03:29:35.993 INFO:tasks.rgw:Creating data pools 2020-11-19T03:29:35.994 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-11-19T03:29:35.994 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-19T03:29:37.241 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'default.rgw.buckets.data' created 2020-11-19T03:29:37.255 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-19T03:29:38.248 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T03:29:38.274 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-19T03:29:39.247 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'default.rgw.buckets.index' created 2020-11-19T03:29:40.680 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-19T03:29:41.260 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T03:29:41.274 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-11-19T03:29:41.275 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-19T03:29:41.579 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-19T03:29:41.590 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-19T03:29:42.262 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T03:29:42.276 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-19T03:29:42.584 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-19T03:29:42.595 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-19T03:29:43.272 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T03:29:43.285 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-11-19T03:29:43.286 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-19T03:29:43.639 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'default.rgw.buckets.data' created 2020-11-19T03:29:43.662 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-19T03:29:44.658 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T03:29:45.463 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-19T03:29:46.664 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'default.rgw.buckets.index' created 2020-11-19T03:29:46.682 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-19T03:29:47.691 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T03:29:47.710 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-11-19T03:29:47.711 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-19T03:29:47.994 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-19T03:29:48.004 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-19T03:29:48.685 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-19T03:29:48.700 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-19T03:29:49.007 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-19T03:29:50.629 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-19T03:29:51.703 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-19T03:29:51.718 DEBUG:tasks.rgw:Pools created 2020-11-19T03:29:51.719 INFO:tasks.rgw:Configuring compression type = random 2020-11-19T03:29:51.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-11-19T03:29:51.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-11-19T03:29:51.720 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-11-19T03:29:51.758 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:29:51.759 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:29:51.770 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:51.765+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.260 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.253+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.260 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.253+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.261 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.257+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.261 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.257+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.261 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.257+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.262 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.257+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.262 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.257+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.278 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.278 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.278 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.279 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.279 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.279 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T03:29:53.279 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.280 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.280 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.285 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.285 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 10 Creating default zonegroup 2020-11-19T03:29:53.285 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 20 get_system_obj_state: rctx=0x7ffeaaa8c6e0 obj=:.pools.avail state=0x564e8afc12e0 s->prefetch_data=0 2020-11-19T03:29:53.286 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.273+0000 7fe385b990c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-19T03:29:53.297 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.293+0000 7fe385b990c0 10 WARNING: store->list_zones() returned r=-2 2020-11-19T03:29:53.298 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.293+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.298 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.293+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.304 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.301+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.305 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.301+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.305 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.301+0000 7fe385b990c0 10 could not read realm id: (2) No such file or directory 2020-11-19T03:29:53.305 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.301+0000 7fe385b990c0 10 WARNING: failed to set zone as default, r=-22 2020-11-19T03:29:53.306 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.301+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.306 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.301+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.312 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.313 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T03:29:53.314 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.315 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:29:53.315 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 zonegroup default 2020-11-19T03:29:53.315 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.316 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:53.316 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 10 Cannot find current period zone using local zone 2020-11-19T03:29:53.316 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 10 Using default name default 2020-11-19T03:29:53.316 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.309+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.317 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.313+0000 7fe385b990c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:29:53.317 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.313+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:53.318 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.313+0000 7fe385b990c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T03:29:53.318 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.313+0000 7fe385b990c0 20 zone default found 2020-11-19T03:29:53.318 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:53.313+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:55.274 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:55.269+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:55.484 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:55.269+0000 7fe385b990c0 20 rados->read ofs=0 len=0 2020-11-19T03:29:55.485 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:55.269+0000 7fe385b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:29:55.485 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:55.269+0000 7fe385b990c0 20 started sync module instance, tier type = 2020-11-19T03:29:55.485 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:55.269+0000 7fe385b990c0 20 started zone id=71f1a900-e9d7-4abc-b574-0f1da8dd78f6 (name=default) with tier type = 2020-11-19T03:29:57.323 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.317+0000 7fe385b990c0 20 add_watcher() i=0 2020-11-19T03:29:57.323 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.317+0000 7fe385b990c0 20 add_watcher() i=1 2020-11-19T03:29:57.323 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.317+0000 7fe385b990c0 20 add_watcher() i=2 2020-11-19T03:29:57.324 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.317+0000 7fe385b990c0 20 add_watcher() i=3 2020-11-19T03:29:57.324 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.317+0000 7fe385b990c0 20 add_watcher() i=4 2020-11-19T03:29:57.324 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.317+0000 7fe385b990c0 20 add_watcher() i=5 2020-11-19T03:29:57.324 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.317+0000 7fe385b990c0 20 add_watcher() i=6 2020-11-19T03:29:57.325 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.321+0000 7fe385b990c0 20 add_watcher() i=7 2020-11-19T03:29:57.325 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.321+0000 7fe385b990c0 2 all 8 watchers are set, enabling cache 2020-11-19T03:29:57.327 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.321+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.327 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.321+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.328 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.321+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.328 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.321+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.328 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.325+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:29:57.330 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.325+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.332 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.325+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:29:57.334 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.329+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.336 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.329+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:29:57.338 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.333+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.340 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.333+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:29:57.343 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.337+0000 7fe385b990c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:29:57.344 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.341+0000 7fe3427e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:29:57.345 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.341+0000 7fe385b990c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:29:57.345 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:29:57.341+0000 7fe385b990c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:01.305 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.301+0000 7fe385b990c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-19T03:30:01.424 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.417+0000 7fe2b17fa700 20 reqs_thread_entry: start 2020-11-19T03:30:01.425 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.417+0000 7fe385b990c0 20 init_complete bucket index max shards: 11 2020-11-19T03:30:01.425 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.417+0000 7fe29effd700 20 reqs_thread_entry: start 2020-11-19T03:30:01.425 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.421+0000 7fe385b990c0 10 Started notification manager with: 1 workers 2020-11-19T03:30:01.425 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.421+0000 7fe29d7fa700 20 INFO: next queues processing will happen at: Thu Nov 19 03:30:31 2020 2020-11-19T03:30:01.426 INFO:teuthology.orchestra.run.smithi066.stderr: 2020-11-19T03:30:01.426 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.421+0000 7fe29cff9700 20 reqs_thread_entry: start 2020-11-19T03:30:01.454 INFO:teuthology.orchestra.run.smithi066.stdout:[] 2020-11-19T03:30:01.455 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.449+0000 7fe385b990c0 20 remove_watcher() i=0 2020-11-19T03:30:01.455 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.449+0000 7fe385b990c0 2 removed watcher, disabling cache 2020-11-19T03:30:01.457 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.453+0000 7fe385b990c0 20 remove_watcher() i=1 2020-11-19T03:30:01.460 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.457+0000 7fe385b990c0 20 remove_watcher() i=2 2020-11-19T03:30:01.463 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.457+0000 7fe385b990c0 20 remove_watcher() i=3 2020-11-19T03:30:01.467 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.461+0000 7fe385b990c0 20 remove_watcher() i=4 2020-11-19T03:30:01.470 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.465+0000 7fe385b990c0 20 remove_watcher() i=5 2020-11-19T03:30:01.473 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.469+0000 7fe385b990c0 20 remove_watcher() i=6 2020-11-19T03:30:01.476 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.469+0000 7fe385b990c0 20 remove_watcher() i=7 2020-11-19T03:30:01.478 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.473+0000 7fe3427e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:30:01.491 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T03:30:01.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:01.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:01.493 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T03:30:01.573 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:01.573 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:01.583 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.577+0000 7fd77290d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:30:01.583 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.577+0000 7fd77290d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:01.590 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.585+0000 7fd77290d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.590 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.585+0000 7fd6a7fff700 20 reqs_thread_entry: start 2020-11-19T03:30:01.591 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.585+0000 7fd77290d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:01.591 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.585+0000 7fd77290d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.592 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.585+0000 7fd77290d0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T03:30:01.592 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.585+0000 7fd77290d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.593 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.589+0000 7fd77290d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.593 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.589+0000 7fd77290d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.594 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.589+0000 7fd77290d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:01.595 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.589+0000 7fd77290d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.595 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.589+0000 7fd77290d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:01.603 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:01.603 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "71f1a900-e9d7-4abc-b574-0f1da8dd78f6", 2020-11-19T03:30:01.603 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "default", 2020-11-19T03:30:01.603 INFO:teuthology.orchestra.run.smithi066.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T03:30:01.604 INFO:teuthology.orchestra.run.smithi066.stdout: "control_pool": "default.rgw.control", 2020-11-19T03:30:01.604 INFO:teuthology.orchestra.run.smithi066.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T03:30:01.604 INFO:teuthology.orchestra.run.smithi066.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T03:30:01.604 INFO:teuthology.orchestra.run.smithi066.stdout: "log_pool": "default.rgw.log", 2020-11-19T03:30:01.604 INFO:teuthology.orchestra.run.smithi066.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T03:30:01.605 INFO:teuthology.orchestra.run.smithi066.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T03:30:01.605 INFO:teuthology.orchestra.run.smithi066.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T03:30:01.605 INFO:teuthology.orchestra.run.smithi066.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T03:30:01.606 INFO:teuthology.orchestra.run.smithi066.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T03:30:01.606 INFO:teuthology.orchestra.run.smithi066.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T03:30:01.606 INFO:teuthology.orchestra.run.smithi066.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T03:30:01.607 INFO:teuthology.orchestra.run.smithi066.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T03:30:01.607 INFO:teuthology.orchestra.run.smithi066.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T03:30:01.607 INFO:teuthology.orchestra.run.smithi066.stdout: "system_key": { 2020-11-19T03:30:01.607 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "", 2020-11-19T03:30:01.607 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "" 2020-11-19T03:30:01.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:01.608 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_pools": [ 2020-11-19T03:30:01.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:01.608 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "default-placement", 2020-11-19T03:30:01.609 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:30:01.609 INFO:teuthology.orchestra.run.smithi066.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T03:30:01.609 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": { 2020-11-19T03:30:01.609 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD": { 2020-11-19T03:30:01.610 INFO:teuthology.orchestra.run.smithi066.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T03:30:01.610 INFO:teuthology.orchestra.run.smithi066.stdout: "compression_type": "random" 2020-11-19T03:30:01.610 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:01.610 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:01.610 INFO:teuthology.orchestra.run.smithi066.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T03:30:01.611 INFO:teuthology.orchestra.run.smithi066.stdout: "index_type": 0 2020-11-19T03:30:01.611 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:01.611 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:01.611 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:01.611 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "", 2020-11-19T03:30:01.612 INFO:teuthology.orchestra.run.smithi066.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T03:30:01.612 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:01.613 DEBUG:tasks.util.rgw: json result: {'id': '71f1a900-e9d7-4abc-b574-0f1da8dd78f6', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T03:30:01.613 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-11-19T03:30:01.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-11-19T03:30:01.614 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-11-19T03:30:01.693 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:01.694 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:01.703 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.697+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.705 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.701+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.705 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.701+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.706 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.701+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.706 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.701+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.707 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.701+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.707 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.701+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.707 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.701+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:01.731 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.725+0000 7f396d57a0c0 20 RGWRados::pool_iterate: got zone_info.71f1a900-e9d7-4abc-b574-0f1da8dd78f6 2020-11-19T03:30:01.731 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.725+0000 7f396d57a0c0 20 RGWRados::pool_iterate: got zonegroup_info.964a6459-dde6-4240-980b-16ac26ccd08e 2020-11-19T03:30:01.732 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.725+0000 7f396d57a0c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-19T03:30:01.732 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.725+0000 7f396d57a0c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-19T03:30:01.732 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.725+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.733 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.725+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:01.733 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.725+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.735 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.729+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T03:30:01.736 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.729+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.736 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:01.737 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.737 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:01.737 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.738 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.739 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T03:30:01.739 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.740 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.740 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.740 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:01.741 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.733+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.741 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:01.742 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 zonegroup default 2020-11-19T03:30:01.742 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.742 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.743 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 10 Cannot find current period zone using local zone 2020-11-19T03:30:01.743 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.743 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.744 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.744 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:01.744 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.737+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.745 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T03:30:01.745 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 zone default found 2020-11-19T03:30:01.745 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.746 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.746 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:01.747 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:01.747 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 started sync module instance, tier type = 2020-11-19T03:30:01.747 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.741+0000 7f396d57a0c0 20 started zone id=71f1a900-e9d7-4abc-b574-0f1da8dd78f6 (name=default) with tier type = 2020-11-19T03:30:01.777 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=0 2020-11-19T03:30:01.777 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=1 2020-11-19T03:30:01.777 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=2 2020-11-19T03:30:01.778 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=3 2020-11-19T03:30:01.778 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=4 2020-11-19T03:30:01.778 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=5 2020-11-19T03:30:01.778 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=6 2020-11-19T03:30:01.779 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 20 add_watcher() i=7 2020-11-19T03:30:01.779 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.773+0000 7f396d57a0c0 2 all 8 watchers are set, enabling cache 2020-11-19T03:30:01.781 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.782 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:01.782 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:01.782 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:01.783 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:01.783 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:01.784 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.784 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:01.784 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.777+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:01.785 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:01.785 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:01.786 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:01.786 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.787 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:01.787 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:01.787 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:01.788 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:01.788 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:01.789 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.781+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.789 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.785+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:01.789 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.785+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:01.790 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.785+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:01.790 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.785+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:01.790 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.785+0000 7f396d57a0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:01.791 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.785+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:01.792 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.785+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.793 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.789+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:01.795 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.789+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.796 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.789+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:01.798 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.793+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.799 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.793+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:01.801 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.797+0000 7f396d57a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:01.802 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.797+0000 7f396d57a0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:30:01.802 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.797+0000 7f396d57a0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:01.803 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.797+0000 7f392a7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:30:01.906 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.897+0000 7f38997fa700 20 reqs_thread_entry: start 2020-11-19T03:30:01.907 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.897+0000 7f3892ffd700 20 reqs_thread_entry: start 2020-11-19T03:30:01.907 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.901+0000 7f396d57a0c0 20 init_complete bucket index max shards: 11 2020-11-19T03:30:01.907 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.901+0000 7f396d57a0c0 10 Started notification manager with: 1 workers 2020-11-19T03:30:01.907 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.901+0000 7f38917fa700 20 INFO: next queues processing will happen at: Thu Nov 19 03:30:32 2020 2020-11-19T03:30:01.907 INFO:teuthology.orchestra.run.smithi066.stderr: 2020-11-19T03:30:01.908 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.901+0000 7f3890ff9700 20 reqs_thread_entry: start 2020-11-19T03:30:01.938 INFO:teuthology.orchestra.run.smithi066.stdout:[] 2020-11-19T03:30:01.938 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.933+0000 7f396d57a0c0 20 remove_watcher() i=0 2020-11-19T03:30:01.938 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.933+0000 7f396d57a0c0 2 removed watcher, disabling cache 2020-11-19T03:30:01.940 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.933+0000 7f396d57a0c0 20 remove_watcher() i=1 2020-11-19T03:30:01.943 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.937+0000 7f396d57a0c0 20 remove_watcher() i=2 2020-11-19T03:30:01.946 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.941+0000 7f396d57a0c0 20 remove_watcher() i=3 2020-11-19T03:30:01.949 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.945+0000 7f396d57a0c0 20 remove_watcher() i=4 2020-11-19T03:30:01.954 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.945+0000 7f396d57a0c0 20 remove_watcher() i=5 2020-11-19T03:30:01.957 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.949+0000 7f396d57a0c0 20 remove_watcher() i=6 2020-11-19T03:30:01.962 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.953+0000 7f396d57a0c0 20 remove_watcher() i=7 2020-11-19T03:30:01.963 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:01.953+0000 7f392a7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:30:01.968 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T03:30:01.970 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:01.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:01.971 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T03:30:02.057 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:02.058 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:02.068 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.061+0000 7f407925b0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:30:02.069 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.061+0000 7f407925b0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:02.078 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f3ff57fa700 20 reqs_thread_entry: start 2020-11-19T03:30:02.080 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:02.084 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:02.084 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:02.084 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T03:30:02.085 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:02.085 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:02.085 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:02.085 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:02.086 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.073+0000 7f407925b0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:02.086 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:02.077+0000 7f407925b0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:02.090 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "71f1a900-e9d7-4abc-b574-0f1da8dd78f6", 2020-11-19T03:30:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "default", 2020-11-19T03:30:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T03:30:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "control_pool": "default.rgw.control", 2020-11-19T03:30:02.091 INFO:teuthology.orchestra.run.smithi066.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T03:30:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T03:30:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "log_pool": "default.rgw.log", 2020-11-19T03:30:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T03:30:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T03:30:02.092 INFO:teuthology.orchestra.run.smithi066.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T03:30:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T03:30:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T03:30:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T03:30:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T03:30:02.093 INFO:teuthology.orchestra.run.smithi066.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T03:30:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T03:30:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "system_key": { 2020-11-19T03:30:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "", 2020-11-19T03:30:02.094 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "" 2020-11-19T03:30:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_pools": [ 2020-11-19T03:30:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "default-placement", 2020-11-19T03:30:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:30:02.095 INFO:teuthology.orchestra.run.smithi066.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T03:30:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": { 2020-11-19T03:30:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD": { 2020-11-19T03:30:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T03:30:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: "compression_type": "random" 2020-11-19T03:30:02.096 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T03:30:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: "index_type": 0 2020-11-19T03:30:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:02.097 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "", 2020-11-19T03:30:02.098 INFO:teuthology.orchestra.run.smithi066.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T03:30:02.098 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:02.099 DEBUG:tasks.util.rgw: json result: {'id': '71f1a900-e9d7-4abc-b574-0f1da8dd78f6', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T03:30:02.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-11-19T03:30:02.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-11-19T03:30:02.100 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-11-19T03:30:02.137 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:02.138 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:02.148 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:02.141+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.656 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.648+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.657 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.648+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.663 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.652+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.664 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.652+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.665 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.652+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.666 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.652+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.667 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.656+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.683 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.672+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.684 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.684 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.691 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.692 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.693 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.694 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T03:30:03.695 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.695 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.696 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.696 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.697 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 10 Creating default zonegroup 2020-11-19T03:30:03.702 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 20 get_system_obj_state: rctx=0x7fffed1a8d70 obj=:.pools.avail state=0x5627ba4202e0 s->prefetch_data=0 2020-11-19T03:30:03.703 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.676+0000 7f38ddf610c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-19T03:30:03.704 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.696+0000 7f38ddf610c0 10 WARNING: store->list_zones() returned r=-2 2020-11-19T03:30:03.704 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.696+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.705 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.696+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.717 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.700+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.718 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.700+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.718 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.700+0000 7f38ddf610c0 10 could not read realm id: (2) No such file or directory 2020-11-19T03:30:03.718 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.700+0000 7f38ddf610c0 10 WARNING: failed to set zone as default, r=-22 2020-11-19T03:30:03.719 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.700+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.719 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.700+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.719 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.719 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T03:30:03.719 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.721 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:03.721 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 zonegroup default 2020-11-19T03:30:03.721 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.721 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:03.721 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 10 Cannot find current period zone using local zone 2020-11-19T03:30:03.722 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 10 Using default name default 2020-11-19T03:30:03.722 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.722 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:03.722 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:03.722 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T03:30:03.723 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 zone default found 2020-11-19T03:30:03.723 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:03.712+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:05.676 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:05.671+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:05.829 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:05.671+0000 7f38ddf610c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:05.830 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:05.671+0000 7f38ddf610c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:05.830 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:05.671+0000 7f38ddf610c0 20 started sync module instance, tier type = 2020-11-19T03:30:05.831 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:05.671+0000 7f38ddf610c0 20 started zone id=33ca7abe-5124-4193-be94-a1a761fec8b9 (name=default) with tier type = 2020-11-19T03:30:07.734 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.727+0000 7f38ddf610c0 20 add_watcher() i=0 2020-11-19T03:30:07.735 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.727+0000 7f38ddf610c0 20 add_watcher() i=1 2020-11-19T03:30:07.735 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.727+0000 7f38ddf610c0 20 add_watcher() i=2 2020-11-19T03:30:07.736 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.727+0000 7f38ddf610c0 20 add_watcher() i=3 2020-11-19T03:30:07.736 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.727+0000 7f38ddf610c0 20 add_watcher() i=4 2020-11-19T03:30:07.736 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.727+0000 7f38ddf610c0 20 add_watcher() i=5 2020-11-19T03:30:07.737 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.727+0000 7f38ddf610c0 20 add_watcher() i=6 2020-11-19T03:30:07.738 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.731+0000 7f38ddf610c0 20 add_watcher() i=7 2020-11-19T03:30:07.738 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.731+0000 7f38ddf610c0 2 all 8 watchers are set, enabling cache 2020-11-19T03:30:07.739 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.735+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.739 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.735+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.740 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.735+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.741 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.735+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.741 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.735+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.739+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.746 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.739+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:07.751 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.747+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.752 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.747+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:07.755 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.751+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.756 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.751+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:07.759 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.755+0000 7f38ddf610c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:07.761 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.755+0000 7f389b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:30:07.762 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.755+0000 7f38ddf610c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:30:07.762 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:07.755+0000 7f38ddf610c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:11.701 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.695+0000 7f38ddf610c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-19T03:30:11.818 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.811+0000 7f380a7fc700 20 reqs_thread_entry: start 2020-11-19T03:30:11.819 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.811+0000 7f38ddf610c0 20 init_complete bucket index max shards: 11 2020-11-19T03:30:11.819 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.811+0000 7f37effff700 20 reqs_thread_entry: start 2020-11-19T03:30:11.819 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.815+0000 7f38ddf610c0 10 Started notification manager with: 1 workers 2020-11-19T03:30:11.819 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.815+0000 7f37ee7fc700 20 INFO: next queues processing will happen at: Thu Nov 19 03:30:42 2020 2020-11-19T03:30:11.820 INFO:teuthology.orchestra.run.smithi154.stderr: 2020-11-19T03:30:11.820 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.815+0000 7f37edffb700 20 reqs_thread_entry: start 2020-11-19T03:30:11.848 INFO:teuthology.orchestra.run.smithi154.stdout:[] 2020-11-19T03:30:11.848 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.843+0000 7f38ddf610c0 20 remove_watcher() i=0 2020-11-19T03:30:11.849 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.843+0000 7f38ddf610c0 2 removed watcher, disabling cache 2020-11-19T03:30:11.850 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.847+0000 7f38ddf610c0 20 remove_watcher() i=1 2020-11-19T03:30:11.852 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.847+0000 7f38ddf610c0 20 remove_watcher() i=2 2020-11-19T03:30:11.856 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.851+0000 7f38ddf610c0 20 remove_watcher() i=3 2020-11-19T03:30:11.860 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.855+0000 7f38ddf610c0 20 remove_watcher() i=4 2020-11-19T03:30:11.863 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.859+0000 7f38ddf610c0 20 remove_watcher() i=5 2020-11-19T03:30:11.865 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.859+0000 7f38ddf610c0 20 remove_watcher() i=6 2020-11-19T03:30:11.868 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.863+0000 7f38ddf610c0 20 remove_watcher() i=7 2020-11-19T03:30:11.870 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.863+0000 7f389b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:30:11.877 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T03:30:11.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:11.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:11.878 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T03:30:11.959 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:11.959 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:11.971 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.967+0000 7f508f48d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:30:11.971 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.967+0000 7f508f48d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:11.976 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.971+0000 7f502cff9700 20 reqs_thread_entry: start 2020-11-19T03:30:11.976 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.971+0000 7f508f48d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:11.980 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.971+0000 7f508f48d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:11.980 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.971+0000 7f508f48d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:11.980 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.975+0000 7f508f48d0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-19T03:30:11.980 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.975+0000 7f508f48d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:11.981 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.975+0000 7f508f48d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:11.981 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.975+0000 7f508f48d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:11.981 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.975+0000 7f508f48d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:11.982 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.975+0000 7f508f48d0c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:11.982 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:11.975+0000 7f508f48d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:11.990 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:30:11.994 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "33ca7abe-5124-4193-be94-a1a761fec8b9", 2020-11-19T03:30:11.995 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "default", 2020-11-19T03:30:11.995 INFO:teuthology.orchestra.run.smithi154.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T03:30:11.995 INFO:teuthology.orchestra.run.smithi154.stdout: "control_pool": "default.rgw.control", 2020-11-19T03:30:11.995 INFO:teuthology.orchestra.run.smithi154.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T03:30:11.996 INFO:teuthology.orchestra.run.smithi154.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T03:30:11.996 INFO:teuthology.orchestra.run.smithi154.stdout: "log_pool": "default.rgw.log", 2020-11-19T03:30:11.996 INFO:teuthology.orchestra.run.smithi154.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T03:30:11.996 INFO:teuthology.orchestra.run.smithi154.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T03:30:11.996 INFO:teuthology.orchestra.run.smithi154.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T03:30:11.997 INFO:teuthology.orchestra.run.smithi154.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T03:30:11.997 INFO:teuthology.orchestra.run.smithi154.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T03:30:11.997 INFO:teuthology.orchestra.run.smithi154.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T03:30:11.997 INFO:teuthology.orchestra.run.smithi154.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T03:30:11.998 INFO:teuthology.orchestra.run.smithi154.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T03:30:11.998 INFO:teuthology.orchestra.run.smithi154.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T03:30:11.998 INFO:teuthology.orchestra.run.smithi154.stdout: "system_key": { 2020-11-19T03:30:11.998 INFO:teuthology.orchestra.run.smithi154.stdout: "access_key": "", 2020-11-19T03:30:11.998 INFO:teuthology.orchestra.run.smithi154.stdout: "secret_key": "" 2020-11-19T03:30:11.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:11.999 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_pools": [ 2020-11-19T03:30:11.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:11.999 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "default-placement", 2020-11-19T03:30:11.999 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:30:12.000 INFO:teuthology.orchestra.run.smithi154.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T03:30:12.000 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": { 2020-11-19T03:30:12.000 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD": { 2020-11-19T03:30:12.000 INFO:teuthology.orchestra.run.smithi154.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T03:30:12.000 INFO:teuthology.orchestra.run.smithi154.stdout: "compression_type": "random" 2020-11-19T03:30:12.001 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:12.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:12.001 INFO:teuthology.orchestra.run.smithi154.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T03:30:12.001 INFO:teuthology.orchestra.run.smithi154.stdout: "index_type": 0 2020-11-19T03:30:12.001 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:12.002 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:12.002 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:12.002 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "", 2020-11-19T03:30:12.002 INFO:teuthology.orchestra.run.smithi154.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T03:30:12.002 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:30:12.003 DEBUG:tasks.util.rgw: json result: {'id': '33ca7abe-5124-4193-be94-a1a761fec8b9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T03:30:12.004 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-11-19T03:30:12.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-11-19T03:30:12.005 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-11-19T03:30:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:12.091 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:12.104 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.099+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.106 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.099+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.106 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.099+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.107 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.103+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.107 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.103+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.107 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.103+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.108 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.103+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.108 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.103+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:12.135 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 RGWRados::pool_iterate: got zone_info.33ca7abe-5124-4193-be94-a1a761fec8b9 2020-11-19T03:30:12.136 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 RGWRados::pool_iterate: got zonegroup_info.8862e737-197e-418d-961c-4bc20ac98a7c 2020-11-19T03:30:12.136 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-19T03:30:12.136 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-19T03:30:12.136 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.137 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:12.137 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.137 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T03:30:12.138 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.138 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:12.138 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.131+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.139 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:12.139 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.140 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.140 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 10 cannot find current period zonegroup using local zonegroup 2020-11-19T03:30:12.140 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.141 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.141 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.141 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:12.142 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.142 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:12.142 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 zonegroup default 2020-11-19T03:30:12.142 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.135+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.143 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.143 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 10 Cannot find current period zone using local zone 2020-11-19T03:30:12.143 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.144 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.144 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.145 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:12.145 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.146 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T03:30:12.146 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 zone default found 2020-11-19T03:30:12.147 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.139+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.147 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.143+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.147 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.143+0000 7f8e630600c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.148 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.143+0000 7f8e630600c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.148 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.143+0000 7f8e630600c0 20 started sync module instance, tier type = 2020-11-19T03:30:12.148 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.143+0000 7f8e630600c0 20 started zone id=33ca7abe-5124-4193-be94-a1a761fec8b9 (name=default) with tier type = 2020-11-19T03:30:12.180 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.175+0000 7f8e630600c0 20 add_watcher() i=0 2020-11-19T03:30:12.180 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.175+0000 7f8e630600c0 20 add_watcher() i=1 2020-11-19T03:30:12.181 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.175+0000 7f8e630600c0 20 add_watcher() i=2 2020-11-19T03:30:12.181 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.175+0000 7f8e630600c0 20 add_watcher() i=3 2020-11-19T03:30:12.181 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.175+0000 7f8e630600c0 20 add_watcher() i=4 2020-11-19T03:30:12.181 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.175+0000 7f8e630600c0 20 add_watcher() i=5 2020-11-19T03:30:12.181 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.175+0000 7f8e630600c0 20 add_watcher() i=6 2020-11-19T03:30:12.184 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.179+0000 7f8e630600c0 20 add_watcher() i=7 2020-11-19T03:30:12.184 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.179+0000 7f8e630600c0 2 all 8 watchers are set, enabling cache 2020-11-19T03:30:12.188 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.183+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.188 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.183+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:12.189 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.183+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:12.189 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.183+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:12.190 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.183+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:12.190 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.183+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:12.190 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.183+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.191 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:12.191 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:12.191 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:12.192 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:12.192 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:12.192 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.193 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:12.193 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:12.193 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:12.194 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:12.194 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:12.194 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.187+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.195 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.191+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-19T03:30:12.195 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.191+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-19T03:30:12.196 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.191+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-19T03:30:12.196 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.191+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-19T03:30:12.196 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.191+0000 7f8e630600c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-19T03:30:12.197 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.191+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:12.198 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.195+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.199 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.195+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:12.202 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.195+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.202 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.199+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:12.205 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.199+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.206 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.199+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-19T03:30:12.208 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.203+0000 7f8e630600c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-19T03:30:12.209 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.203+0000 7f8e1ffe7700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:30:12.209 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.203+0000 7f8e630600c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:30:12.210 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.203+0000 7f8e630600c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:12.304 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.299+0000 7f8d8effd700 20 reqs_thread_entry: start 2020-11-19T03:30:12.305 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.299+0000 7f8d8cff9700 20 reqs_thread_entry: start 2020-11-19T03:30:12.305 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.299+0000 7f8e630600c0 20 init_complete bucket index max shards: 11 2020-11-19T03:30:12.305 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.299+0000 7f8e630600c0 10 Started notification manager with: 1 workers 2020-11-19T03:30:12.305 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.299+0000 7f8d82ffd700 20 INFO: next queues processing will happen at: Thu Nov 19 03:30:42 2020 2020-11-19T03:30:12.306 INFO:teuthology.orchestra.run.smithi154.stderr: 2020-11-19T03:30:12.306 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.299+0000 7f8d827fc700 20 reqs_thread_entry: start 2020-11-19T03:30:12.331 INFO:teuthology.orchestra.run.smithi154.stdout:[] 2020-11-19T03:30:12.331 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.327+0000 7f8e630600c0 20 remove_watcher() i=0 2020-11-19T03:30:12.331 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.327+0000 7f8e630600c0 2 removed watcher, disabling cache 2020-11-19T03:30:12.333 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.327+0000 7f8e630600c0 20 remove_watcher() i=1 2020-11-19T03:30:12.336 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.331+0000 7f8e630600c0 20 remove_watcher() i=2 2020-11-19T03:30:12.339 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.335+0000 7f8e630600c0 20 remove_watcher() i=3 2020-11-19T03:30:12.342 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.335+0000 7f8e630600c0 20 remove_watcher() i=4 2020-11-19T03:30:12.345 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.339+0000 7f8e630600c0 20 remove_watcher() i=5 2020-11-19T03:30:12.347 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.343+0000 7f8e630600c0 20 remove_watcher() i=6 2020-11-19T03:30:12.350 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.347+0000 7f8e630600c0 20 remove_watcher() i=7 2020-11-19T03:30:12.352 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.347+0000 7f8e1ffe7700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-19T03:30:12.358 DEBUG:tasks.util.rgw: json result: [] 2020-11-19T03:30:12.359 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:12.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-19T03:30:12.360 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-19T03:30:12.448 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:12.449 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:12.462 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.455+0000 7f1ce89510c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-19T03:30:12.463 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.455+0000 7f1ce89510c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-19T03:30:12.467 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1c8cff9700 20 reqs_thread_entry: start 2020-11-19T03:30:12.468 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1ce89510c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.469 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1ce89510c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:12.469 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1ce89510c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.470 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1ce89510c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-19T03:30:12.470 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1ce89510c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.470 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1ce89510c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-19T03:30:12.471 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.463+0000 7f1ce89510c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.471 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.467+0000 7f1ce89510c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-19T03:30:12.471 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.467+0000 7f1ce89510c0 20 rados->read ofs=0 len=0 2020-11-19T03:30:12.472 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:12.467+0000 7f1ce89510c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-19T03:30:12.479 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:30:12.479 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "33ca7abe-5124-4193-be94-a1a761fec8b9", 2020-11-19T03:30:12.480 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "default", 2020-11-19T03:30:12.480 INFO:teuthology.orchestra.run.smithi154.stdout: "domain_root": "default.rgw.meta:root", 2020-11-19T03:30:12.480 INFO:teuthology.orchestra.run.smithi154.stdout: "control_pool": "default.rgw.control", 2020-11-19T03:30:12.480 INFO:teuthology.orchestra.run.smithi154.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-19T03:30:12.481 INFO:teuthology.orchestra.run.smithi154.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-19T03:30:12.481 INFO:teuthology.orchestra.run.smithi154.stdout: "log_pool": "default.rgw.log", 2020-11-19T03:30:12.481 INFO:teuthology.orchestra.run.smithi154.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-19T03:30:12.481 INFO:teuthology.orchestra.run.smithi154.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-19T03:30:12.481 INFO:teuthology.orchestra.run.smithi154.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-19T03:30:12.482 INFO:teuthology.orchestra.run.smithi154.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-19T03:30:12.482 INFO:teuthology.orchestra.run.smithi154.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-19T03:30:12.482 INFO:teuthology.orchestra.run.smithi154.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-19T03:30:12.482 INFO:teuthology.orchestra.run.smithi154.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-19T03:30:12.483 INFO:teuthology.orchestra.run.smithi154.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-19T03:30:12.483 INFO:teuthology.orchestra.run.smithi154.stdout: "otp_pool": "default.rgw.otp", 2020-11-19T03:30:12.483 INFO:teuthology.orchestra.run.smithi154.stdout: "system_key": { 2020-11-19T03:30:12.483 INFO:teuthology.orchestra.run.smithi154.stdout: "access_key": "", 2020-11-19T03:30:12.483 INFO:teuthology.orchestra.run.smithi154.stdout: "secret_key": "" 2020-11-19T03:30:12.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:12.484 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_pools": [ 2020-11-19T03:30:12.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:12.484 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "default-placement", 2020-11-19T03:30:12.485 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:30:12.485 INFO:teuthology.orchestra.run.smithi154.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-19T03:30:12.485 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": { 2020-11-19T03:30:12.485 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD": { 2020-11-19T03:30:12.485 INFO:teuthology.orchestra.run.smithi154.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-19T03:30:12.486 INFO:teuthology.orchestra.run.smithi154.stdout: "compression_type": "random" 2020-11-19T03:30:12.486 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:12.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:12.486 INFO:teuthology.orchestra.run.smithi154.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-19T03:30:12.486 INFO:teuthology.orchestra.run.smithi154.stdout: "index_type": 0 2020-11-19T03:30:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "", 2020-11-19T03:30:12.488 INFO:teuthology.orchestra.run.smithi154.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-19T03:30:12.488 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:30:12.489 DEBUG:tasks.util.rgw: json result: {'id': '33ca7abe-5124-4193-be94-a1a761fec8b9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-19T03:30:12.490 INFO:tasks.rgw:Starting rgw... 2020-11-19T03:30:12.490 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-11-19T03:30:12.490 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-19T03:30:12.491 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-19T03:30:12.492 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-19T03:30:12.495 INFO:tasks.rgw.c1.client.0:Started 2020-11-19T03:30:12.495 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-11-19T03:30:12.495 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-19T03:30:12.496 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-19T03:30:12.496 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-19T03:30:12.500 INFO:tasks.rgw.c1.client.1:Started 2020-11-19T03:30:12.500 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-11-19T03:30:12.501 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-19T03:30:12.501 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-19T03:30:12.501 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-19T03:30:12.543 INFO:tasks.rgw.c2.client.0:Started 2020-11-19T03:30:12.543 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-11-19T03:30:12.544 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-19T03:30:12.544 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-19T03:30:12.545 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-19T03:30:12.549 INFO:tasks.rgw.c2.client.1:Started 2020-11-19T03:30:12.550 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi066.front.sepia.ceph.com:8000/ 2020-11-19T03:30:12.551 INFO:teuthology.orchestra.run.smithi066:> curl http://smithi066.front.sepia.ceph.com:8000/ 2020-11-19T03:30:12.587 INFO:teuthology.orchestra.run.smithi066.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:12.588 INFO:teuthology.orchestra.run.smithi066.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:12.591 INFO:teuthology.orchestra.run.smithi066.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi066.front.sepia.ceph.com port 8000: Connection refused 2020-11-19T03:30:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T03:30:13.593 INFO:teuthology.orchestra.run.smithi066:> curl http://smithi066.front.sepia.ceph.com:8000/ 2020-11-19T03:30:13.609 INFO:teuthology.orchestra.run.smithi066.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:13.609 INFO:teuthology.orchestra.run.smithi066.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:13.615 INFO:teuthology.orchestra.run.smithi066.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T03:30:13.616 INFO:teuthology.orchestra.run.smithi066.stdout:anonymous 2020-11-19T03:30:13.616 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi066.front.sepia.ceph.com:8001/ 2020-11-19T03:30:13.616 INFO:teuthology.orchestra.run.smithi066:> curl http://smithi066.front.sepia.ceph.com:8001/ 2020-11-19T03:30:13.682 INFO:teuthology.orchestra.run.smithi066.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:13.683 INFO:teuthology.orchestra.run.smithi066.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:13.685 INFO:teuthology.orchestra.run.smithi066.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-19T03:30:13.686 INFO:teuthology.orchestra.run.smithi066.stdout:anonymous 2020-11-19T03:30:13.687 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi154.front.sepia.ceph.com:8000/ 2020-11-19T03:30:13.687 INFO:teuthology.orchestra.run.smithi154:> curl http://smithi154.front.sepia.ceph.com:8000/ 2020-11-19T03:30:13.719 INFO:teuthology.orchestra.run.smithi154.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:13.720 INFO:teuthology.orchestra.run.smithi154.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:13.727 INFO:teuthology.orchestra.run.smithi154.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-19T03:30:13.729 INFO:teuthology.orchestra.run.smithi154.stdout:anonymous 2020-11-19T03:30:13.729 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi154.front.sepia.ceph.com:8001/ 2020-11-19T03:30:13.730 INFO:teuthology.orchestra.run.smithi154:> curl http://smithi154.front.sepia.ceph.com:8001/ 2020-11-19T03:30:13.754 INFO:teuthology.orchestra.run.smithi154.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:13.754 INFO:teuthology.orchestra.run.smithi154.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:13.756 INFO:teuthology.orchestra.run.smithi154.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T03:30:13.760 INFO:teuthology.orchestra.run.smithi154.stdout:anonymous 2020-11-19T03:30:13.760 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-11-19T03:30:14.305 INFO:tasks.rgw_multisite:creating realm.. 2020-11-19T03:30:15.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:15.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:15.895 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:15.935 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:15.936 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:15.989 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:15.990 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:15.990 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-realm", 2020-11-19T03:30:15.990 INFO:teuthology.orchestra.run.smithi066.stdout: "current_period": "05cd2172-571f-487d-af1d-f00f9436c7f5", 2020-11-19T03:30:15.990 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 1 2020-11-19T03:30:15.990 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:15.991 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-11-19T03:30:15.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:15.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:15.993 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:16.070 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:16.070 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:16.092 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:16.085+0000 7fcf29bef0c0 -1 old format 2020-11-19T03:30:16.108 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:16.108 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "test-zonegroup", 2020-11-19T03:30:16.108 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zonegroup", 2020-11-19T03:30:16.109 INFO:teuthology.orchestra.run.smithi066.stdout: "api_name": "", 2020-11-19T03:30:16.109 INFO:teuthology.orchestra.run.smithi066.stdout: "is_master": "true", 2020-11-19T03:30:16.109 INFO:teuthology.orchestra.run.smithi066.stdout: "endpoints": [ 2020-11-19T03:30:16.109 INFO:teuthology.orchestra.run.smithi066.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:30:16.110 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:16.110 INFO:teuthology.orchestra.run.smithi066.stdout: "hostnames": [], 2020-11-19T03:30:16.110 INFO:teuthology.orchestra.run.smithi066.stdout: "hostnames_s3website": [], 2020-11-19T03:30:16.110 INFO:teuthology.orchestra.run.smithi066.stdout: "master_zone": "", 2020-11-19T03:30:16.110 INFO:teuthology.orchestra.run.smithi066.stdout: "zones": [], 2020-11-19T03:30:16.111 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_targets": [], 2020-11-19T03:30:16.111 INFO:teuthology.orchestra.run.smithi066.stdout: "default_placement": "", 2020-11-19T03:30:16.111 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:16.111 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_policy": { 2020-11-19T03:30:16.111 INFO:teuthology.orchestra.run.smithi066.stdout: "groups": [] 2020-11-19T03:30:16.112 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:16.112 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:16.113 INFO:tasks.rgw_multisite:creating master zone.. 2020-11-19T03:30:16.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi066.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:16.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi066.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:16.114 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --endpoints http://smithi066.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:16.204 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:16.204 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:16.275 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:16.275 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:16.275 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone1", 2020-11-19T03:30:16.276 INFO:teuthology.orchestra.run.smithi066.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-19T03:30:16.276 INFO:teuthology.orchestra.run.smithi066.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-19T03:30:16.276 INFO:teuthology.orchestra.run.smithi066.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-19T03:30:16.276 INFO:teuthology.orchestra.run.smithi066.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-19T03:30:16.276 INFO:teuthology.orchestra.run.smithi066.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-19T03:30:16.277 INFO:teuthology.orchestra.run.smithi066.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-19T03:30:16.277 INFO:teuthology.orchestra.run.smithi066.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-19T03:30:16.277 INFO:teuthology.orchestra.run.smithi066.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-19T03:30:16.277 INFO:teuthology.orchestra.run.smithi066.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-19T03:30:16.277 INFO:teuthology.orchestra.run.smithi066.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-19T03:30:16.278 INFO:teuthology.orchestra.run.smithi066.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-19T03:30:16.278 INFO:teuthology.orchestra.run.smithi066.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-19T03:30:16.278 INFO:teuthology.orchestra.run.smithi066.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-19T03:30:16.278 INFO:teuthology.orchestra.run.smithi066.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-19T03:30:16.279 INFO:teuthology.orchestra.run.smithi066.stdout: "system_key": { 2020-11-19T03:30:16.279 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:30:16.279 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:30:16.279 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:16.279 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_pools": [ 2020-11-19T03:30:16.280 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:16.280 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "default-placement", 2020-11-19T03:30:16.280 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:30:16.280 INFO:teuthology.orchestra.run.smithi066.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-19T03:30:16.280 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": { 2020-11-19T03:30:16.281 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD": { 2020-11-19T03:30:16.281 INFO:teuthology.orchestra.run.smithi066.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-11-19T03:30:16.281 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:16.281 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:16.281 INFO:teuthology.orchestra.run.smithi066.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-19T03:30:16.282 INFO:teuthology.orchestra.run.smithi066.stdout: "index_type": 0 2020-11-19T03:30:16.282 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:16.282 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:16.282 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:16.282 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:16.283 INFO:teuthology.orchestra.run.smithi066.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-19T03:30:16.283 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:16.284 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-11-19T03:30:17.248 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-11-19T03:30:17.271 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-11-19T03:30:18.256 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-11-19T03:30:18.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:18.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:18.285 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:18.329 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:18.329 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:18.351 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone1", 2020-11-19T03:30:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-19T03:30:18.352 INFO:teuthology.orchestra.run.smithi066.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-19T03:30:18.352 INFO:teuthology.orchestra.run.smithi066.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-19T03:30:18.352 INFO:teuthology.orchestra.run.smithi066.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-19T03:30:18.352 INFO:teuthology.orchestra.run.smithi066.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-19T03:30:18.352 INFO:teuthology.orchestra.run.smithi066.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-19T03:30:18.353 INFO:teuthology.orchestra.run.smithi066.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-19T03:30:18.353 INFO:teuthology.orchestra.run.smithi066.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-19T03:30:18.353 INFO:teuthology.orchestra.run.smithi066.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-19T03:30:18.353 INFO:teuthology.orchestra.run.smithi066.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-19T03:30:18.353 INFO:teuthology.orchestra.run.smithi066.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-19T03:30:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-19T03:30:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-19T03:30:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-19T03:30:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "system_key": { 2020-11-19T03:30:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:30:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:30:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_pools": [ 2020-11-19T03:30:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "default-placement", 2020-11-19T03:30:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:30:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-19T03:30:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": { 2020-11-19T03:30:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD": { 2020-11-19T03:30:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-19T03:30:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "compression_type": "random" 2020-11-19T03:30:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-19T03:30:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "index_type": 0 2020-11-19T03:30:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "", 2020-11-19T03:30:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-19T03:30:18.359 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:18.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:18.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:18.361 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:18.444 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:18.444 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:26.538 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:26.539 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:30:26.539 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 1, 2020-11-19T03:30:26.539 INFO:teuthology.orchestra.run.smithi066.stdout: "predecessor_uuid": "05cd2172-571f-487d-af1d-f00f9436c7f5", 2020-11-19T03:30:26.539 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": [], 2020-11-19T03:30:26.540 INFO:teuthology.orchestra.run.smithi066.stdout: "period_map": { 2020-11-19T03:30:26.540 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:30:26.540 INFO:teuthology.orchestra.run.smithi066.stdout: "zonegroups": [ 2020-11-19T03:30:26.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:26.540 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "test-zonegroup", 2020-11-19T03:30:26.541 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zonegroup", 2020-11-19T03:30:26.541 INFO:teuthology.orchestra.run.smithi066.stdout: "api_name": "", 2020-11-19T03:30:26.541 INFO:teuthology.orchestra.run.smithi066.stdout: "is_master": "true", 2020-11-19T03:30:26.541 INFO:teuthology.orchestra.run.smithi066.stdout: "endpoints": [ 2020-11-19T03:30:26.542 INFO:teuthology.orchestra.run.smithi066.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:30:26.542 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:26.542 INFO:teuthology.orchestra.run.smithi066.stdout: "hostnames": [], 2020-11-19T03:30:26.542 INFO:teuthology.orchestra.run.smithi066.stdout: "hostnames_s3website": [], 2020-11-19T03:30:26.542 INFO:teuthology.orchestra.run.smithi066.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:26.543 INFO:teuthology.orchestra.run.smithi066.stdout: "zones": [ 2020-11-19T03:30:26.543 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:26.543 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:26.543 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone1", 2020-11-19T03:30:26.543 INFO:teuthology.orchestra.run.smithi066.stdout: "endpoints": [ 2020-11-19T03:30:26.544 INFO:teuthology.orchestra.run.smithi066.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:30:26.544 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:26.544 INFO:teuthology.orchestra.run.smithi066.stdout: "log_meta": "false", 2020-11-19T03:30:26.544 INFO:teuthology.orchestra.run.smithi066.stdout: "log_data": "false", 2020-11-19T03:30:26.544 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:30:26.545 INFO:teuthology.orchestra.run.smithi066.stdout: "read_only": "false", 2020-11-19T03:30:26.545 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_type": "", 2020-11-19T03:30:26.545 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from_all": "true", 2020-11-19T03:30:26.545 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from": [], 2020-11-19T03:30:26.546 INFO:teuthology.orchestra.run.smithi066.stdout: "redirect_zone": "" 2020-11-19T03:30:26.546 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:26.546 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:26.546 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_targets": [ 2020-11-19T03:30:26.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:26.547 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "default-placement", 2020-11-19T03:30:26.547 INFO:teuthology.orchestra.run.smithi066.stdout: "tags": [], 2020-11-19T03:30:26.548 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": [ 2020-11-19T03:30:26.548 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD" 2020-11-19T03:30:26.548 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:30:26.549 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:26.549 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:26.549 INFO:teuthology.orchestra.run.smithi066.stdout: "default_placement": "default-placement", 2020-11-19T03:30:26.549 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:26.550 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_policy": { 2020-11-19T03:30:26.550 INFO:teuthology.orchestra.run.smithi066.stdout: "groups": [] 2020-11-19T03:30:26.550 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:26.550 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:26.551 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:26.551 INFO:teuthology.orchestra.run.smithi066.stdout: "short_zone_ids": [ 2020-11-19T03:30:26.551 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:26.551 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:26.552 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 3678712233 2020-11-19T03:30:26.552 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:26.552 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:30:26.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:26.553 INFO:teuthology.orchestra.run.smithi066.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:30:26.553 INFO:teuthology.orchestra.run.smithi066.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:26.553 INFO:teuthology.orchestra.run.smithi066.stdout: "period_config": { 2020-11-19T03:30:26.553 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_quota": { 2020-11-19T03:30:26.553 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:30:26.554 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:30:26.554 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:30:26.554 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:30:26.554 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:30:26.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:26.555 INFO:teuthology.orchestra.run.smithi066.stdout: "user_quota": { 2020-11-19T03:30:26.555 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:30:26.555 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:30:26.555 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:30:26.555 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:30:26.556 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:30:26.556 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:26.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:26.556 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:26.557 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_name": "test-realm", 2020-11-19T03:30:26.557 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:30:26.557 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:26.559 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-11-19T03:30:26.559 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T03:30:26.604 INFO:tasks.rgw.c1.client.0.smithi066.stdout:2020-11-19T03:30:26.597+0000 7fa0dcc87700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 13174) UID: 0 2020-11-19T03:30:26.604 INFO:tasks.rgw.c1.client.0.smithi066.stdout:2020-11-19T03:30:26.597+0000 7fa0f3f26100 -1 shutting down 2020-11-19T03:30:29.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-11-19T03:30:32.660 INFO:tasks.rgw.c1.client.0:Stopped 2020-11-19T03:30:32.661 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-19T03:30:32.662 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-19T03:30:32.675 INFO:tasks.rgw.c1.client.0:Started 2020-11-19T03:30:32.676 INFO:teuthology.orchestra.run.smithi066:> curl http://smithi066.front.sepia.ceph.com:8000 2020-11-19T03:30:32.693 INFO:teuthology.orchestra.run.smithi066.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:32.693 INFO:teuthology.orchestra.run.smithi066.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:32.696 INFO:teuthology.orchestra.run.smithi066.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi066.front.sepia.ceph.com port 8000: Connection refused 2020-11-19T03:30:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T03:30:33.699 INFO:teuthology.orchestra.run.smithi066:> curl http://smithi066.front.sepia.ceph.com:8000 2020-11-19T03:30:33.719 INFO:teuthology.orchestra.run.smithi066.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:33.719 INFO:teuthology.orchestra.run.smithi066.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:33.726 INFO:teuthology.orchestra.run.smithi066.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T03:30:33.727 INFO:teuthology.orchestra.run.smithi066.stdout:anonymous 2020-11-19T03:30:33.727 INFO:tasks.rgw_multisite:creating admin user.. 2020-11-19T03:30:33.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:33.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:33.729 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:33.823 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:33.823 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:34.046 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:35.683 INFO:teuthology.orchestra.run.smithi066.stdout: "user_id": "realm-admin", 2020-11-19T03:30:35.684 INFO:teuthology.orchestra.run.smithi066.stdout: "display_name": "Realm Admin", 2020-11-19T03:30:35.684 INFO:teuthology.orchestra.run.smithi066.stdout: "email": "", 2020-11-19T03:30:35.684 INFO:teuthology.orchestra.run.smithi066.stdout: "suspended": 0, 2020-11-19T03:30:35.684 INFO:teuthology.orchestra.run.smithi066.stdout: "max_buckets": 1000, 2020-11-19T03:30:35.685 INFO:teuthology.orchestra.run.smithi066.stdout: "subusers": [], 2020-11-19T03:30:35.685 INFO:teuthology.orchestra.run.smithi066.stdout: "keys": [ 2020-11-19T03:30:35.685 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:35.685 INFO:teuthology.orchestra.run.smithi066.stdout: "user": "realm-admin", 2020-11-19T03:30:35.686 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:30:35.686 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:30:35.686 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:35.686 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:35.686 INFO:teuthology.orchestra.run.smithi066.stdout: "swift_keys": [], 2020-11-19T03:30:35.687 INFO:teuthology.orchestra.run.smithi066.stdout: "caps": [], 2020-11-19T03:30:35.687 INFO:teuthology.orchestra.run.smithi066.stdout: "op_mask": "read, write, delete", 2020-11-19T03:30:35.687 INFO:teuthology.orchestra.run.smithi066.stdout: "system": "true", 2020-11-19T03:30:35.687 INFO:teuthology.orchestra.run.smithi066.stdout: "default_placement": "", 2020-11-19T03:30:35.687 INFO:teuthology.orchestra.run.smithi066.stdout: "default_storage_class": "", 2020-11-19T03:30:35.688 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_tags": [], 2020-11-19T03:30:35.688 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_quota": { 2020-11-19T03:30:35.688 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:30:35.688 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:30:35.688 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:30:35.689 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:30:35.689 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:30:35.689 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:35.689 INFO:teuthology.orchestra.run.smithi066.stdout: "user_quota": { 2020-11-19T03:30:35.689 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:30:35.690 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:30:35.690 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:30:35.690 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:30:35.690 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:30:35.690 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:35.691 INFO:teuthology.orchestra.run.smithi066.stdout: "temp_url_keys": [], 2020-11-19T03:30:35.691 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "rgw", 2020-11-19T03:30:35.691 INFO:teuthology.orchestra.run.smithi066.stdout: "mfa_ids": [] 2020-11-19T03:30:35.691 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:35.691 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:30:35.693 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-19T03:30:35.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi066.front.sepia.ceph.com:8000', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:35.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi066.front.sepia.ceph.com:8000', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:35.694 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi066.front.sepia.ceph.com:8000 --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:35.735 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:35.735 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:35.851 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:30:35.851 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:35.852 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-realm", 2020-11-19T03:30:35.852 INFO:teuthology.orchestra.run.smithi154.stdout: "current_period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:30:35.852 INFO:teuthology.orchestra.run.smithi154.stdout: "epoch": 2 2020-11-19T03:30:35.853 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:30:35.854 INFO:tasks.rgw_multisite:creating zone.. 2020-11-19T03:30:35.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi154.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:35.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi154.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:35.855 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --endpoints http://smithi154.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:35.941 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:35.941 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:36.008 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:36.003+0000 7f3b6b97a0c0 0 failed reading obj info from .rgw.root:zone_info.e1cea8b6-8c08-4879-ac34-c5e52e428237: (2) No such file or directory 2020-11-19T03:30:36.009 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:30:36.003+0000 7f3b6b97a0c0 0 WARNING: could not read zone params for zone id=e1cea8b6-8c08-4879-ac34-c5e52e428237 name=test-zone1 2020-11-19T03:30:36.019 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:30:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:30:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone2", 2020-11-19T03:30:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T03:30:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T03:30:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T03:30:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T03:30:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T03:30:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T03:30:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T03:30:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T03:30:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T03:30:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T03:30:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T03:30:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T03:30:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T03:30:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T03:30:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "system_key": { 2020-11-19T03:30:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:30:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:30:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_pools": [ 2020-11-19T03:30:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "default-placement", 2020-11-19T03:30:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:30:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T03:30:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": { 2020-11-19T03:30:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD": { 2020-11-19T03:30:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-11-19T03:30:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T03:30:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "index_type": 0 2020-11-19T03:30:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T03:30:36.028 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:30:36.030 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-11-19T03:30:36.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-11-19T03:30:36.675 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-11-19T03:30:36.701 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-11-19T03:30:37.698 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-11-19T03:30:37.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:37.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:37.719 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:37.770 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:37.770 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:37.793 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:30:37.794 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:30:37.795 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone2", 2020-11-19T03:30:37.795 INFO:teuthology.orchestra.run.smithi154.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-19T03:30:37.797 INFO:teuthology.orchestra.run.smithi154.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-19T03:30:37.798 INFO:teuthology.orchestra.run.smithi154.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-19T03:30:37.798 INFO:teuthology.orchestra.run.smithi154.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-19T03:30:37.799 INFO:teuthology.orchestra.run.smithi154.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-19T03:30:37.799 INFO:teuthology.orchestra.run.smithi154.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-19T03:30:37.799 INFO:teuthology.orchestra.run.smithi154.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-19T03:30:37.799 INFO:teuthology.orchestra.run.smithi154.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-19T03:30:37.799 INFO:teuthology.orchestra.run.smithi154.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-19T03:30:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-19T03:30:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-19T03:30:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-19T03:30:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-19T03:30:37.801 INFO:teuthology.orchestra.run.smithi154.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-19T03:30:37.801 INFO:teuthology.orchestra.run.smithi154.stdout: "system_key": { 2020-11-19T03:30:37.801 INFO:teuthology.orchestra.run.smithi154.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:30:37.801 INFO:teuthology.orchestra.run.smithi154.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:30:37.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:37.802 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_pools": [ 2020-11-19T03:30:37.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:37.802 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "default-placement", 2020-11-19T03:30:37.802 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:30:37.802 INFO:teuthology.orchestra.run.smithi154.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-19T03:30:37.803 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": { 2020-11-19T03:30:37.803 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD": { 2020-11-19T03:30:37.803 INFO:teuthology.orchestra.run.smithi154.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-19T03:30:37.803 INFO:teuthology.orchestra.run.smithi154.stdout: "compression_type": "random" 2020-11-19T03:30:37.803 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:37.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:37.804 INFO:teuthology.orchestra.run.smithi154.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-19T03:30:37.804 INFO:teuthology.orchestra.run.smithi154.stdout: "index_type": 0 2020-11-19T03:30:37.804 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:37.804 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:37.805 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:37.805 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "", 2020-11-19T03:30:37.805 INFO:teuthology.orchestra.run.smithi154.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-19T03:30:37.805 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:30:37.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:37.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:37.807 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:37.889 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:37.889 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:45.882 INFO:teuthology.orchestra.run.smithi154.stderr:Sending period to new master zone e1cea8b6-8c08-4879-ac34-c5e52e428237 2020-11-19T03:30:45.976 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:30:45.976 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:30:45.976 INFO:teuthology.orchestra.run.smithi154.stdout: "epoch": 2, 2020-11-19T03:30:45.977 INFO:teuthology.orchestra.run.smithi154.stdout: "predecessor_uuid": "05cd2172-571f-487d-af1d-f00f9436c7f5", 2020-11-19T03:30:45.977 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": [], 2020-11-19T03:30:45.977 INFO:teuthology.orchestra.run.smithi154.stdout: "period_map": { 2020-11-19T03:30:45.977 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:30:45.978 INFO:teuthology.orchestra.run.smithi154.stdout: "zonegroups": [ 2020-11-19T03:30:45.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:45.978 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "test-zonegroup", 2020-11-19T03:30:45.978 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zonegroup", 2020-11-19T03:30:45.978 INFO:teuthology.orchestra.run.smithi154.stdout: "api_name": "", 2020-11-19T03:30:45.979 INFO:teuthology.orchestra.run.smithi154.stdout: "is_master": "true", 2020-11-19T03:30:45.979 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:30:45.979 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:30:45.979 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:45.979 INFO:teuthology.orchestra.run.smithi154.stdout: "hostnames": [], 2020-11-19T03:30:45.980 INFO:teuthology.orchestra.run.smithi154.stdout: "hostnames_s3website": [], 2020-11-19T03:30:45.980 INFO:teuthology.orchestra.run.smithi154.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:45.980 INFO:teuthology.orchestra.run.smithi154.stdout: "zones": [ 2020-11-19T03:30:45.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:45.980 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:30:45.981 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone2", 2020-11-19T03:30:45.981 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:30:45.981 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi154.front.sepia.ceph.com:8000" 2020-11-19T03:30:45.981 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:45.982 INFO:teuthology.orchestra.run.smithi154.stdout: "log_meta": "false", 2020-11-19T03:30:45.982 INFO:teuthology.orchestra.run.smithi154.stdout: "log_data": "true", 2020-11-19T03:30:45.982 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:30:45.982 INFO:teuthology.orchestra.run.smithi154.stdout: "read_only": "false", 2020-11-19T03:30:45.982 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_type": "", 2020-11-19T03:30:45.983 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from_all": "true", 2020-11-19T03:30:45.983 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from": [], 2020-11-19T03:30:45.983 INFO:teuthology.orchestra.run.smithi154.stdout: "redirect_zone": "" 2020-11-19T03:30:45.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:45.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:45.984 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:45.984 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone1", 2020-11-19T03:30:45.984 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:30:45.985 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:30:45.985 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:45.985 INFO:teuthology.orchestra.run.smithi154.stdout: "log_meta": "false", 2020-11-19T03:30:45.985 INFO:teuthology.orchestra.run.smithi154.stdout: "log_data": "true", 2020-11-19T03:30:45.985 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:30:45.986 INFO:teuthology.orchestra.run.smithi154.stdout: "read_only": "false", 2020-11-19T03:30:45.986 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_type": "", 2020-11-19T03:30:45.986 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from_all": "true", 2020-11-19T03:30:45.986 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from": [], 2020-11-19T03:30:45.986 INFO:teuthology.orchestra.run.smithi154.stdout: "redirect_zone": "" 2020-11-19T03:30:45.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:45.987 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:45.987 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_targets": [ 2020-11-19T03:30:45.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:45.988 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "default-placement", 2020-11-19T03:30:45.988 INFO:teuthology.orchestra.run.smithi154.stdout: "tags": [], 2020-11-19T03:30:45.988 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": [ 2020-11-19T03:30:45.988 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD" 2020-11-19T03:30:45.988 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:30:45.989 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:45.989 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:45.989 INFO:teuthology.orchestra.run.smithi154.stdout: "default_placement": "default-placement", 2020-11-19T03:30:45.989 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:45.990 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_policy": { 2020-11-19T03:30:45.990 INFO:teuthology.orchestra.run.smithi154.stdout: "groups": [] 2020-11-19T03:30:45.991 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:45.991 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:45.991 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:30:45.991 INFO:teuthology.orchestra.run.smithi154.stdout: "short_zone_ids": [ 2020-11-19T03:30:45.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:45.992 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:30:45.992 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 2905304302 2020-11-19T03:30:45.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:45.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:30:45.993 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:45.993 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 3678712233 2020-11-19T03:30:45.993 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:45.993 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:30:45.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:45.994 INFO:teuthology.orchestra.run.smithi154.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:30:45.994 INFO:teuthology.orchestra.run.smithi154.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:30:45.994 INFO:teuthology.orchestra.run.smithi154.stdout: "period_config": { 2020-11-19T03:30:45.994 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_quota": { 2020-11-19T03:30:45.994 INFO:teuthology.orchestra.run.smithi154.stdout: "enabled": false, 2020-11-19T03:30:45.995 INFO:teuthology.orchestra.run.smithi154.stdout: "check_on_raw": false, 2020-11-19T03:30:45.995 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size": -1, 2020-11-19T03:30:45.995 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size_kb": 0, 2020-11-19T03:30:45.995 INFO:teuthology.orchestra.run.smithi154.stdout: "max_objects": -1 2020-11-19T03:30:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: "user_quota": { 2020-11-19T03:30:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: "enabled": false, 2020-11-19T03:30:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: "check_on_raw": false, 2020-11-19T03:30:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size": -1, 2020-11-19T03:30:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size_kb": 0, 2020-11-19T03:30:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: "max_objects": -1 2020-11-19T03:30:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:30:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:30:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:45.998 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_name": "test-realm", 2020-11-19T03:30:45.998 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:30:45.998 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:30:46.000 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-11-19T03:30:46.001 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T03:30:46.042 INFO:tasks.rgw.c2.client.0.smithi154.stdout:2020-11-19T03:30:46.039+0000 7feb38aaa700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 13409) UID: 0 2020-11-19T03:30:46.043 INFO:tasks.rgw.c2.client.0.smithi154.stdout:2020-11-19T03:30:46.039+0000 7feb5054a100 -1 shutting down 2020-11-19T03:30:47.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-11-19T03:30:52.102 INFO:tasks.rgw.c2.client.0:Stopped 2020-11-19T03:30:52.102 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-19T03:30:52.103 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-19T03:30:52.106 INFO:tasks.rgw.c2.client.0:Started 2020-11-19T03:30:52.106 INFO:teuthology.orchestra.run.smithi154:> curl http://smithi154.front.sepia.ceph.com:8000 2020-11-19T03:30:52.131 INFO:teuthology.orchestra.run.smithi154.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:52.133 INFO:teuthology.orchestra.run.smithi154.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:52.133 INFO:teuthology.orchestra.run.smithi154.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi154.front.sepia.ceph.com port 8000: Connection refused 2020-11-19T03:30:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T03:30:53.138 INFO:teuthology.orchestra.run.smithi154:> curl http://smithi154.front.sepia.ceph.com:8000 2020-11-19T03:30:53.166 INFO:teuthology.orchestra.run.smithi154.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:30:53.166 INFO:teuthology.orchestra.run.smithi154.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:30:53.173 INFO:teuthology.orchestra.run.smithi154.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T03:30:53.174 INFO:teuthology.orchestra.run.smithi154.stdout:anonymous 2020-11-19T03:30:53.175 INFO:tasks.rgw_multisite:creating zone.. 2020-11-19T03:30:53.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi066.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:53.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi066.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:53.176 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --endpoints http://smithi066.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:53.223 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:53.224 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:53.307 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:53.301+0000 7f4dc97cf0c0 0 failed reading obj info from .rgw.root:zone_info.22a854ca-7cbc-4f7c-a17c-c7423da02f47: (2) No such file or directory 2020-11-19T03:30:53.308 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:30:53.301+0000 7f4dc97cf0c0 0 WARNING: could not read zone params for zone id=22a854ca-7cbc-4f7c-a17c-c7423da02f47 name=test-zone2 2020-11-19T03:30:53.314 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:53.315 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "a6b0bdf9-e42c-4a6b-8842-528c2cda2174", 2020-11-19T03:30:53.315 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone3", 2020-11-19T03:30:53.316 INFO:teuthology.orchestra.run.smithi066.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-19T03:30:53.316 INFO:teuthology.orchestra.run.smithi066.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-19T03:30:53.316 INFO:teuthology.orchestra.run.smithi066.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-19T03:30:53.317 INFO:teuthology.orchestra.run.smithi066.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-19T03:30:53.317 INFO:teuthology.orchestra.run.smithi066.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-19T03:30:53.317 INFO:teuthology.orchestra.run.smithi066.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-19T03:30:53.318 INFO:teuthology.orchestra.run.smithi066.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-19T03:30:53.318 INFO:teuthology.orchestra.run.smithi066.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-19T03:30:53.318 INFO:teuthology.orchestra.run.smithi066.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-19T03:30:53.319 INFO:teuthology.orchestra.run.smithi066.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-19T03:30:53.319 INFO:teuthology.orchestra.run.smithi066.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-19T03:30:53.320 INFO:teuthology.orchestra.run.smithi066.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-19T03:30:53.321 INFO:teuthology.orchestra.run.smithi066.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-19T03:30:53.322 INFO:teuthology.orchestra.run.smithi066.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-19T03:30:53.322 INFO:teuthology.orchestra.run.smithi066.stdout: "system_key": { 2020-11-19T03:30:53.323 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:30:53.323 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:30:53.323 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:53.324 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_pools": [ 2020-11-19T03:30:53.324 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:53.325 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "default-placement", 2020-11-19T03:30:53.325 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:30:53.325 INFO:teuthology.orchestra.run.smithi066.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-19T03:30:53.326 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": { 2020-11-19T03:30:53.326 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD": { 2020-11-19T03:30:53.326 INFO:teuthology.orchestra.run.smithi066.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-11-19T03:30:53.327 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:53.328 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:53.328 INFO:teuthology.orchestra.run.smithi066.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-19T03:30:53.329 INFO:teuthology.orchestra.run.smithi066.stdout: "index_type": 0 2020-11-19T03:30:53.329 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:53.329 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:53.330 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:53.330 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:30:53.330 INFO:teuthology.orchestra.run.smithi066.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-19T03:30:53.330 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:53.333 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-11-19T03:30:53.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-11-19T03:30:54.297 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-11-19T03:30:54.322 INFO:teuthology.orchestra.run.smithi066:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-11-19T03:30:55.299 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-11-19T03:30:55.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:55.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:55.336 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:55.378 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:55.379 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:30:55.404 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:30:55.405 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "a6b0bdf9-e42c-4a6b-8842-528c2cda2174", 2020-11-19T03:30:55.405 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone3", 2020-11-19T03:30:55.405 INFO:teuthology.orchestra.run.smithi066.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-19T03:30:55.405 INFO:teuthology.orchestra.run.smithi066.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-19T03:30:55.406 INFO:teuthology.orchestra.run.smithi066.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-19T03:30:55.406 INFO:teuthology.orchestra.run.smithi066.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-19T03:30:55.406 INFO:teuthology.orchestra.run.smithi066.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-19T03:30:55.406 INFO:teuthology.orchestra.run.smithi066.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-19T03:30:55.406 INFO:teuthology.orchestra.run.smithi066.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-19T03:30:55.407 INFO:teuthology.orchestra.run.smithi066.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-19T03:30:55.407 INFO:teuthology.orchestra.run.smithi066.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-19T03:30:55.407 INFO:teuthology.orchestra.run.smithi066.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-19T03:30:55.407 INFO:teuthology.orchestra.run.smithi066.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-19T03:30:55.407 INFO:teuthology.orchestra.run.smithi066.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-19T03:30:55.408 INFO:teuthology.orchestra.run.smithi066.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-19T03:30:55.408 INFO:teuthology.orchestra.run.smithi066.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-19T03:30:55.408 INFO:teuthology.orchestra.run.smithi066.stdout: "system_key": { 2020-11-19T03:30:55.408 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:30:55.408 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:30:55.409 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:55.409 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_pools": [ 2020-11-19T03:30:55.409 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:30:55.409 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "default-placement", 2020-11-19T03:30:55.410 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:30:55.410 INFO:teuthology.orchestra.run.smithi066.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-19T03:30:55.410 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": { 2020-11-19T03:30:55.410 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD": { 2020-11-19T03:30:55.410 INFO:teuthology.orchestra.run.smithi066.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-11-19T03:30:55.411 INFO:teuthology.orchestra.run.smithi066.stdout: "compression_type": "random" 2020-11-19T03:30:55.411 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:55.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:30:55.411 INFO:teuthology.orchestra.run.smithi066.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-19T03:30:55.412 INFO:teuthology.orchestra.run.smithi066.stdout: "index_type": 0 2020-11-19T03:30:55.412 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:55.412 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:30:55.412 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:30:55.412 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "", 2020-11-19T03:30:55.413 INFO:teuthology.orchestra.run.smithi066.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-19T03:30:55.413 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:30:55.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:55.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:30:55.415 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:30:55.493 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:30:55.494 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:03.495 INFO:teuthology.orchestra.run.smithi066.stderr:Sending period to new master zone e1cea8b6-8c08-4879-ac34-c5e52e428237 2020-11-19T03:31:03.556 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:31:03.557 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:03.557 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 3, 2020-11-19T03:31:03.557 INFO:teuthology.orchestra.run.smithi066.stdout: "predecessor_uuid": "05cd2172-571f-487d-af1d-f00f9436c7f5", 2020-11-19T03:31:03.557 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": [], 2020-11-19T03:31:03.558 INFO:teuthology.orchestra.run.smithi066.stdout: "period_map": { 2020-11-19T03:31:03.558 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:03.558 INFO:teuthology.orchestra.run.smithi066.stdout: "zonegroups": [ 2020-11-19T03:31:03.558 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.558 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "test-zonegroup", 2020-11-19T03:31:03.559 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zonegroup", 2020-11-19T03:31:03.559 INFO:teuthology.orchestra.run.smithi066.stdout: "api_name": "", 2020-11-19T03:31:03.559 INFO:teuthology.orchestra.run.smithi066.stdout: "is_master": "true", 2020-11-19T03:31:03.559 INFO:teuthology.orchestra.run.smithi066.stdout: "endpoints": [ 2020-11-19T03:31:03.559 INFO:teuthology.orchestra.run.smithi066.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:31:03.560 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:03.560 INFO:teuthology.orchestra.run.smithi066.stdout: "hostnames": [], 2020-11-19T03:31:03.560 INFO:teuthology.orchestra.run.smithi066.stdout: "hostnames_s3website": [], 2020-11-19T03:31:03.560 INFO:teuthology.orchestra.run.smithi066.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:03.560 INFO:teuthology.orchestra.run.smithi066.stdout: "zones": [ 2020-11-19T03:31:03.561 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.561 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:31:03.561 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone2", 2020-11-19T03:31:03.561 INFO:teuthology.orchestra.run.smithi066.stdout: "endpoints": [ 2020-11-19T03:31:03.562 INFO:teuthology.orchestra.run.smithi066.stdout: "http://smithi154.front.sepia.ceph.com:8000" 2020-11-19T03:31:03.562 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:03.562 INFO:teuthology.orchestra.run.smithi066.stdout: "log_meta": "false", 2020-11-19T03:31:03.562 INFO:teuthology.orchestra.run.smithi066.stdout: "log_data": "true", 2020-11-19T03:31:03.562 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:31:03.563 INFO:teuthology.orchestra.run.smithi066.stdout: "read_only": "false", 2020-11-19T03:31:03.563 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_type": "", 2020-11-19T03:31:03.563 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from_all": "true", 2020-11-19T03:31:03.563 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from": [], 2020-11-19T03:31:03.563 INFO:teuthology.orchestra.run.smithi066.stdout: "redirect_zone": "" 2020-11-19T03:31:03.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:03.564 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.564 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "a6b0bdf9-e42c-4a6b-8842-528c2cda2174", 2020-11-19T03:31:03.564 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone3", 2020-11-19T03:31:03.564 INFO:teuthology.orchestra.run.smithi066.stdout: "endpoints": [ 2020-11-19T03:31:03.565 INFO:teuthology.orchestra.run.smithi066.stdout: "http://smithi066.front.sepia.ceph.com:8001" 2020-11-19T03:31:03.565 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:03.565 INFO:teuthology.orchestra.run.smithi066.stdout: "log_meta": "false", 2020-11-19T03:31:03.565 INFO:teuthology.orchestra.run.smithi066.stdout: "log_data": "true", 2020-11-19T03:31:03.565 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:31:03.566 INFO:teuthology.orchestra.run.smithi066.stdout: "read_only": "false", 2020-11-19T03:31:03.566 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_type": "", 2020-11-19T03:31:03.566 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from_all": "true", 2020-11-19T03:31:03.566 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from": [], 2020-11-19T03:31:03.566 INFO:teuthology.orchestra.run.smithi066.stdout: "redirect_zone": "" 2020-11-19T03:31:03.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:03.567 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.567 INFO:teuthology.orchestra.run.smithi066.stdout: "id": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:03.567 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "test-zone1", 2020-11-19T03:31:03.567 INFO:teuthology.orchestra.run.smithi066.stdout: "endpoints": [ 2020-11-19T03:31:03.568 INFO:teuthology.orchestra.run.smithi066.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:31:03.568 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:03.568 INFO:teuthology.orchestra.run.smithi066.stdout: "log_meta": "false", 2020-11-19T03:31:03.568 INFO:teuthology.orchestra.run.smithi066.stdout: "log_data": "true", 2020-11-19T03:31:03.568 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:31:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "read_only": "false", 2020-11-19T03:31:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_type": "", 2020-11-19T03:31:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from_all": "true", 2020-11-19T03:31:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_from": [], 2020-11-19T03:31:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "redirect_zone": "" 2020-11-19T03:31:03.570 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:03.570 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:03.570 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_targets": [ 2020-11-19T03:31:03.570 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.570 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "default-placement", 2020-11-19T03:31:03.571 INFO:teuthology.orchestra.run.smithi066.stdout: "tags": [], 2020-11-19T03:31:03.571 INFO:teuthology.orchestra.run.smithi066.stdout: "storage_classes": [ 2020-11-19T03:31:03.571 INFO:teuthology.orchestra.run.smithi066.stdout: "STANDARD" 2020-11-19T03:31:03.571 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:31:03.571 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:03.572 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:03.572 INFO:teuthology.orchestra.run.smithi066.stdout: "default_placement": "default-placement", 2020-11-19T03:31:03.572 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:31:03.572 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_policy": { 2020-11-19T03:31:03.572 INFO:teuthology.orchestra.run.smithi066.stdout: "groups": [] 2020-11-19T03:31:03.573 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:03.573 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:03.573 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:03.573 INFO:teuthology.orchestra.run.smithi066.stdout: "short_zone_ids": [ 2020-11-19T03:31:03.573 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.574 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:31:03.574 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 2905304302 2020-11-19T03:31:03.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:03.574 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.574 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "a6b0bdf9-e42c-4a6b-8842-528c2cda2174", 2020-11-19T03:31:03.575 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 2385389576 2020-11-19T03:31:03.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:03.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:03.575 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:03.575 INFO:teuthology.orchestra.run.smithi066.stdout: "val": 3678712233 2020-11-19T03:31:03.576 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:03.576 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:31:03.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:03.576 INFO:teuthology.orchestra.run.smithi066.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:31:03.576 INFO:teuthology.orchestra.run.smithi066.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:03.577 INFO:teuthology.orchestra.run.smithi066.stdout: "period_config": { 2020-11-19T03:31:03.577 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_quota": { 2020-11-19T03:31:03.577 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:31:03.577 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:31:03.577 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:31:03.578 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:31:03.578 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:31:03.578 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:03.578 INFO:teuthology.orchestra.run.smithi066.stdout: "user_quota": { 2020-11-19T03:31:03.578 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:31:03.579 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:31:03.582 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:31:03.582 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:31:03.582 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:31:03.583 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:03.583 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:03.583 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:31:03.583 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_name": "test-realm", 2020-11-19T03:31:03.583 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:31:03.584 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:31:03.585 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-19T03:31:03.585 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T03:31:03.634 INFO:tasks.rgw.c1.client.1.smithi066.stdout:2020-11-19T03:31:03.629+0000 7f281d503700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 13176) UID: 0 2020-11-19T03:31:03.635 INFO:tasks.rgw.c1.client.1.smithi066.stdout:2020-11-19T03:31:03.629+0000 7f28347a2100 -1 shutting down 2020-11-19T03:31:04.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-19T03:31:09.687 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-19T03:31:09.687 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-19T03:31:09.688 INFO:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-19T03:31:09.693 INFO:tasks.rgw.c1.client.1:Started 2020-11-19T03:31:09.694 INFO:teuthology.orchestra.run.smithi066:> curl http://smithi066.front.sepia.ceph.com:8001 2020-11-19T03:31:09.714 INFO:teuthology.orchestra.run.smithi066.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:31:09.715 INFO:teuthology.orchestra.run.smithi066.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:31:09.720 INFO:teuthology.orchestra.run.smithi066.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi066.front.sepia.ceph.com port 8001: Connection refused 2020-11-19T03:31:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T03:31:10.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-19T03:31:10.723 INFO:teuthology.orchestra.run.smithi066:> curl http://smithi066.front.sepia.ceph.com:8001 2020-11-19T03:31:10.740 INFO:teuthology.orchestra.run.smithi066.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:31:10.740 INFO:teuthology.orchestra.run.smithi066.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:31:10.744 INFO:teuthology.orchestra.run.smithi066.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-19T03:31:10.745 INFO:teuthology.orchestra.run.smithi066.stdout:anonymous 2020-11-19T03:31:10.745 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-19T03:31:10.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi066.front.sepia.ceph.com:8000', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--url', 'http://smithi066.front.sepia.ceph.com:8000', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:10.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi066.front.sepia.ceph.com:8000', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--url', 'http://smithi066.front.sepia.ceph.com:8000', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:10.747 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi066.front.sepia.ceph.com:8000 --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --url http://smithi066.front.sepia.ceph.com:8000 --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:31:10.795 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:10.795 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:10.836 INFO:tasks.rgw.c2.client.0.smithi154.stdout:2020-11-19T03:31:10.831+0000 7f95fee15700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.a75027ec-0548-42c4-a73d-b2228086d344.control err=-107 2020-11-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:31:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-realm", 2020-11-19T03:31:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "current_period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "epoch": 2 2020-11-19T03:31:10.864 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:10.865 INFO:tasks.rgw_multisite:creating zone.. 2020-11-19T03:31:10.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi154.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:10.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JK5XCCRIB9ILKKV7', '--secret', '281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7', '--endpoints', 'http://smithi154.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:10.871 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key JK5XCCRIB9ILKKV7 --secret 281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7 --endpoints http://smithi154.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:31:10.921 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:10.921 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:10.997 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:31:10.995+0000 7f57a97890c0 0 failed reading obj info from .rgw.root:zone_info.a6b0bdf9-e42c-4a6b-8842-528c2cda2174: (2) No such file or directory 2020-11-19T03:31:10.997 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:31:10.995+0000 7f57a97890c0 0 WARNING: could not read zone params for zone id=a6b0bdf9-e42c-4a6b-8842-528c2cda2174 name=test-zone3 2020-11-19T03:31:10.998 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:31:10.995+0000 7f57a97890c0 0 failed reading obj info from .rgw.root:zone_info.e1cea8b6-8c08-4879-ac34-c5e52e428237: (2) No such file or directory 2020-11-19T03:31:10.998 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:31:10.995+0000 7f57a97890c0 0 WARNING: could not read zone params for zone id=e1cea8b6-8c08-4879-ac34-c5e52e428237 name=test-zone1 2020-11-19T03:31:11.003 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:11.003 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "30ca04d4-c6e6-457f-b4cf-fa39cbb412b6", 2020-11-19T03:31:11.004 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone4", 2020-11-19T03:31:11.004 INFO:teuthology.orchestra.run.smithi154.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-19T03:31:11.004 INFO:teuthology.orchestra.run.smithi154.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-19T03:31:11.005 INFO:teuthology.orchestra.run.smithi154.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-19T03:31:11.005 INFO:teuthology.orchestra.run.smithi154.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-19T03:31:11.005 INFO:teuthology.orchestra.run.smithi154.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-19T03:31:11.005 INFO:teuthology.orchestra.run.smithi154.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-19T03:31:11.005 INFO:teuthology.orchestra.run.smithi154.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-19T03:31:11.006 INFO:teuthology.orchestra.run.smithi154.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-19T03:31:11.006 INFO:teuthology.orchestra.run.smithi154.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-19T03:31:11.006 INFO:teuthology.orchestra.run.smithi154.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-19T03:31:11.006 INFO:teuthology.orchestra.run.smithi154.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-19T03:31:11.006 INFO:teuthology.orchestra.run.smithi154.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-19T03:31:11.007 INFO:teuthology.orchestra.run.smithi154.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-19T03:31:11.007 INFO:teuthology.orchestra.run.smithi154.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-19T03:31:11.007 INFO:teuthology.orchestra.run.smithi154.stdout: "system_key": { 2020-11-19T03:31:11.007 INFO:teuthology.orchestra.run.smithi154.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:31:11.007 INFO:teuthology.orchestra.run.smithi154.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:31:11.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:11.008 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_pools": [ 2020-11-19T03:31:11.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:11.008 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "default-placement", 2020-11-19T03:31:11.009 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:11.009 INFO:teuthology.orchestra.run.smithi154.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-19T03:31:11.009 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": { 2020-11-19T03:31:11.009 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD": { 2020-11-19T03:31:11.009 INFO:teuthology.orchestra.run.smithi154.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-11-19T03:31:11.010 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:11.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:11.010 INFO:teuthology.orchestra.run.smithi154.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-19T03:31:11.010 INFO:teuthology.orchestra.run.smithi154.stdout: "index_type": 0 2020-11-19T03:31:11.010 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:11.011 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:11.011 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:11.011 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_config": { 2020-11-19T03:31:11.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event_retention_days": 7, 2020-11-19T03:31:11.011 INFO:teuthology.orchestra.run.smithi154.stdout: "start_with_full_sync": false 2020-11-19T03:31:11.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:11.012 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:31:11.012 INFO:teuthology.orchestra.run.smithi154.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-19T03:31:11.012 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:11.013 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-11-19T03:31:11.681 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-11-19T03:31:11.703 INFO:teuthology.orchestra.run.smithi154:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-11-19T03:31:12.694 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-11-19T03:31:12.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:12.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:12.726 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:31:12.772 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:12.773 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:12.797 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "30ca04d4-c6e6-457f-b4cf-fa39cbb412b6", 2020-11-19T03:31:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone4", 2020-11-19T03:31:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-19T03:31:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-19T03:31:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-19T03:31:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-19T03:31:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-19T03:31:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-19T03:31:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-19T03:31:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-19T03:31:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-19T03:31:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-19T03:31:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-19T03:31:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-19T03:31:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-19T03:31:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-19T03:31:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "system_key": { 2020-11-19T03:31:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "access_key": "JK5XCCRIB9ILKKV7", 2020-11-19T03:31:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "secret_key": "281ZTYarmWhN5eTJqrNPLKNcWwzf4TH7" 2020-11-19T03:31:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_pools": [ 2020-11-19T03:31:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "default-placement", 2020-11-19T03:31:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-19T03:31:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": { 2020-11-19T03:31:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD": { 2020-11-19T03:31:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-11-19T03:31:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "compression_type": "random" 2020-11-19T03:31:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-19T03:31:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "index_type": 0 2020-11-19T03:31:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_config": { 2020-11-19T03:31:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event_retention_days": 7, 2020-11-19T03:31:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "start_with_full_sync": false 2020-11-19T03:31:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "", 2020-11-19T03:31:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-19T03:31:12.821 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:12.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:12.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:12.825 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-19T03:31:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:20.887 INFO:teuthology.orchestra.run.smithi154.stderr:Sending period to new master zone e1cea8b6-8c08-4879-ac34-c5e52e428237 2020-11-19T03:31:20.958 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:20.959 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:20.959 INFO:teuthology.orchestra.run.smithi154.stdout: "epoch": 4, 2020-11-19T03:31:20.959 INFO:teuthology.orchestra.run.smithi154.stdout: "predecessor_uuid": "05cd2172-571f-487d-af1d-f00f9436c7f5", 2020-11-19T03:31:20.959 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": [], 2020-11-19T03:31:20.960 INFO:teuthology.orchestra.run.smithi154.stdout: "period_map": { 2020-11-19T03:31:20.960 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:20.960 INFO:teuthology.orchestra.run.smithi154.stdout: "zonegroups": [ 2020-11-19T03:31:20.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:20.960 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "test-zonegroup", 2020-11-19T03:31:20.961 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zonegroup", 2020-11-19T03:31:20.961 INFO:teuthology.orchestra.run.smithi154.stdout: "api_name": "", 2020-11-19T03:31:20.961 INFO:teuthology.orchestra.run.smithi154.stdout: "is_master": "true", 2020-11-19T03:31:20.961 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:31:20.962 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:31:20.962 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.962 INFO:teuthology.orchestra.run.smithi154.stdout: "hostnames": [], 2020-11-19T03:31:20.962 INFO:teuthology.orchestra.run.smithi154.stdout: "hostnames_s3website": [], 2020-11-19T03:31:20.962 INFO:teuthology.orchestra.run.smithi154.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:20.963 INFO:teuthology.orchestra.run.smithi154.stdout: "zones": [ 2020-11-19T03:31:20.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:20.963 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:31:20.963 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone2", 2020-11-19T03:31:20.963 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:31:20.964 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi154.front.sepia.ceph.com:8000" 2020-11-19T03:31:20.964 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.964 INFO:teuthology.orchestra.run.smithi154.stdout: "log_meta": "false", 2020-11-19T03:31:20.964 INFO:teuthology.orchestra.run.smithi154.stdout: "log_data": "true", 2020-11-19T03:31:20.964 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:31:20.965 INFO:teuthology.orchestra.run.smithi154.stdout: "read_only": "false", 2020-11-19T03:31:20.965 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_type": "", 2020-11-19T03:31:20.965 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from_all": "true", 2020-11-19T03:31:20.965 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from": [], 2020-11-19T03:31:20.965 INFO:teuthology.orchestra.run.smithi154.stdout: "redirect_zone": "" 2020-11-19T03:31:20.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:20.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:20.966 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "30ca04d4-c6e6-457f-b4cf-fa39cbb412b6", 2020-11-19T03:31:20.966 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone4", 2020-11-19T03:31:20.967 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:31:20.967 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi154.front.sepia.ceph.com:8001" 2020-11-19T03:31:20.967 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.967 INFO:teuthology.orchestra.run.smithi154.stdout: "log_meta": "false", 2020-11-19T03:31:20.967 INFO:teuthology.orchestra.run.smithi154.stdout: "log_data": "true", 2020-11-19T03:31:20.968 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:31:20.968 INFO:teuthology.orchestra.run.smithi154.stdout: "read_only": "false", 2020-11-19T03:31:20.968 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_type": "pubsub", 2020-11-19T03:31:20.968 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from_all": "false", 2020-11-19T03:31:20.968 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from": [ 2020-11-19T03:31:20.969 INFO:teuthology.orchestra.run.smithi154.stdout: "test-zone1" 2020-11-19T03:31:20.969 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.969 INFO:teuthology.orchestra.run.smithi154.stdout: "redirect_zone": "" 2020-11-19T03:31:20.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:20.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:20.970 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "a6b0bdf9-e42c-4a6b-8842-528c2cda2174", 2020-11-19T03:31:20.970 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone3", 2020-11-19T03:31:20.971 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:31:20.971 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi066.front.sepia.ceph.com:8001" 2020-11-19T03:31:20.971 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.971 INFO:teuthology.orchestra.run.smithi154.stdout: "log_meta": "false", 2020-11-19T03:31:20.972 INFO:teuthology.orchestra.run.smithi154.stdout: "log_data": "true", 2020-11-19T03:31:20.972 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:31:20.972 INFO:teuthology.orchestra.run.smithi154.stdout: "read_only": "false", 2020-11-19T03:31:20.972 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_type": "", 2020-11-19T03:31:20.972 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from_all": "true", 2020-11-19T03:31:20.973 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from": [], 2020-11-19T03:31:20.973 INFO:teuthology.orchestra.run.smithi154.stdout: "redirect_zone": "" 2020-11-19T03:31:20.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:20.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:20.973 INFO:teuthology.orchestra.run.smithi154.stdout: "id": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:20.974 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "test-zone1", 2020-11-19T03:31:20.974 INFO:teuthology.orchestra.run.smithi154.stdout: "endpoints": [ 2020-11-19T03:31:20.974 INFO:teuthology.orchestra.run.smithi154.stdout: "http://smithi066.front.sepia.ceph.com:8000" 2020-11-19T03:31:20.974 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.974 INFO:teuthology.orchestra.run.smithi154.stdout: "log_meta": "false", 2020-11-19T03:31:20.975 INFO:teuthology.orchestra.run.smithi154.stdout: "log_data": "true", 2020-11-19T03:31:20.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_index_max_shards": 11, 2020-11-19T03:31:20.975 INFO:teuthology.orchestra.run.smithi154.stdout: "read_only": "false", 2020-11-19T03:31:20.975 INFO:teuthology.orchestra.run.smithi154.stdout: "tier_type": "", 2020-11-19T03:31:20.975 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from_all": "true", 2020-11-19T03:31:20.976 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_from": [], 2020-11-19T03:31:20.976 INFO:teuthology.orchestra.run.smithi154.stdout: "redirect_zone": "" 2020-11-19T03:31:20.976 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:20.976 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.976 INFO:teuthology.orchestra.run.smithi154.stdout: "placement_targets": [ 2020-11-19T03:31:20.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:20.977 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "default-placement", 2020-11-19T03:31:20.977 INFO:teuthology.orchestra.run.smithi154.stdout: "tags": [], 2020-11-19T03:31:20.977 INFO:teuthology.orchestra.run.smithi154.stdout: "storage_classes": [ 2020-11-19T03:31:20.977 INFO:teuthology.orchestra.run.smithi154.stdout: "STANDARD" 2020-11-19T03:31:20.978 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:20.978 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:20.978 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.979 INFO:teuthology.orchestra.run.smithi154.stdout: "default_placement": "default-placement", 2020-11-19T03:31:20.979 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:31:20.980 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_policy": { 2020-11-19T03:31:20.980 INFO:teuthology.orchestra.run.smithi154.stdout: "groups": [] 2020-11-19T03:31:20.980 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:20.980 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:20.981 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2020-11-19T03:31:20.981 INFO:teuthology.orchestra.run.smithi154.stdout: "short_zone_ids": [ 2020-11-19T03:31:20.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:21.058 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "22a854ca-7cbc-4f7c-a17c-c7423da02f47", 2020-11-19T03:31:21.059 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 2905304302 2020-11-19T03:31:21.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:21.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:21.060 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "30ca04d4-c6e6-457f-b4cf-fa39cbb412b6", 2020-11-19T03:31:21.060 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 3990935180 2020-11-19T03:31:21.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:21.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:21.060 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "a6b0bdf9-e42c-4a6b-8842-528c2cda2174", 2020-11-19T03:31:21.061 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 2385389576 2020-11-19T03:31:21.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:21.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:21.061 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:21.062 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 3678712233 2020-11-19T03:31:21.062 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:21.062 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:21.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:21.062 INFO:teuthology.orchestra.run.smithi154.stdout: "master_zonegroup": "test-zonegroup", 2020-11-19T03:31:21.063 INFO:teuthology.orchestra.run.smithi154.stdout: "master_zone": "e1cea8b6-8c08-4879-ac34-c5e52e428237", 2020-11-19T03:31:21.063 INFO:teuthology.orchestra.run.smithi154.stdout: "period_config": { 2020-11-19T03:31:21.063 INFO:teuthology.orchestra.run.smithi154.stdout: "bucket_quota": { 2020-11-19T03:31:21.063 INFO:teuthology.orchestra.run.smithi154.stdout: "enabled": false, 2020-11-19T03:31:21.063 INFO:teuthology.orchestra.run.smithi154.stdout: "check_on_raw": false, 2020-11-19T03:31:21.064 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size": -1, 2020-11-19T03:31:21.064 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size_kb": 0, 2020-11-19T03:31:21.064 INFO:teuthology.orchestra.run.smithi154.stdout: "max_objects": -1 2020-11-19T03:31:21.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:21.064 INFO:teuthology.orchestra.run.smithi154.stdout: "user_quota": { 2020-11-19T03:31:21.065 INFO:teuthology.orchestra.run.smithi154.stdout: "enabled": false, 2020-11-19T03:31:21.065 INFO:teuthology.orchestra.run.smithi154.stdout: "check_on_raw": false, 2020-11-19T03:31:21.065 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size": -1, 2020-11-19T03:31:21.065 INFO:teuthology.orchestra.run.smithi154.stdout: "max_size_kb": 0, 2020-11-19T03:31:21.065 INFO:teuthology.orchestra.run.smithi154.stdout: "max_objects": -1 2020-11-19T03:31:21.066 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:21.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:21.066 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_id": "a75027ec-0548-42c4-a73d-b2228086d344", 2020-11-19T03:31:21.066 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_name": "test-realm", 2020-11-19T03:31:21.066 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:21.067 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:21.068 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-11-19T03:31:21.068 INFO:teuthology.orchestra.run:waiting for 300 2020-11-19T03:31:21.109 INFO:tasks.rgw.c2.client.1.smithi154.stdout:2020-11-19T03:31:21.107+0000 7f407ffff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 13411) UID: 0 2020-11-19T03:31:21.110 INFO:tasks.rgw.c2.client.1.smithi154.stdout:2020-11-19T03:31:21.107+0000 7f40a3b4a100 -1 shutting down 2020-11-19T03:31:22.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-11-19T03:31:27.170 INFO:tasks.rgw.c2.client.1:Stopped 2020-11-19T03:31:27.170 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-19T03:31:27.171 INFO:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-19T03:31:27.178 INFO:tasks.rgw.c2.client.1:Started 2020-11-19T03:31:27.178 INFO:teuthology.orchestra.run.smithi154:> curl http://smithi154.front.sepia.ceph.com:8001 2020-11-19T03:31:27.196 INFO:teuthology.orchestra.run.smithi154.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:31:27.197 INFO:teuthology.orchestra.run.smithi154.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:31:27.200 INFO:teuthology.orchestra.run.smithi154.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi154.front.sepia.ceph.com port 8001: Connection refused 2020-11-19T03:31:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-19T03:31:27.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-11-19T03:31:28.203 INFO:teuthology.orchestra.run.smithi154:> curl http://smithi154.front.sepia.ceph.com:8001 2020-11-19T03:31:28.221 INFO:teuthology.orchestra.run.smithi154.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-19T03:31:28.222 INFO:teuthology.orchestra.run.smithi154.stderr: Dload Upload Total Spent Left Speed 2020-11-19T03:31:28.225 INFO:teuthology.orchestra.run.smithi154.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2020-11-19T03:31:28.225 INFO:teuthology.orchestra.run.smithi154.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005fb5e710-10e6-test-zone4","HostId":"10e6-test-zone4-test-zonegroup"} 2020-11-19T03:31:28.226 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-11-19T03:31:28.226 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-11-19T03:31:28.324 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-11-19T03:31:28.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:28.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-19T03:31:28.326 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-19T03:31:28.379 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:28.379 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:28.629 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:31:28.630 INFO:teuthology.orchestra.run.smithi066.stdout: "user_id": "rgw-multisite-test-user", 2020-11-19T03:31:28.630 INFO:teuthology.orchestra.run.smithi066.stdout: "display_name": "Multisite Test User", 2020-11-19T03:31:28.630 INFO:teuthology.orchestra.run.smithi066.stdout: "email": "", 2020-11-19T03:31:28.630 INFO:teuthology.orchestra.run.smithi066.stdout: "suspended": 0, 2020-11-19T03:31:28.631 INFO:teuthology.orchestra.run.smithi066.stdout: "max_buckets": 1000, 2020-11-19T03:31:28.631 INFO:teuthology.orchestra.run.smithi066.stdout: "subusers": [], 2020-11-19T03:31:28.631 INFO:teuthology.orchestra.run.smithi066.stdout: "keys": [ 2020-11-19T03:31:28.631 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:28.632 INFO:teuthology.orchestra.run.smithi066.stdout: "user": "rgw-multisite-test-user", 2020-11-19T03:31:28.632 INFO:teuthology.orchestra.run.smithi066.stdout: "access_key": "0DJ8ZCEIXS3JL95GHLVV", 2020-11-19T03:31:28.632 INFO:teuthology.orchestra.run.smithi066.stdout: "secret_key": "2PKBXcUnAiSzhXo9dOl3tj09nHTsrXKcwTvE05YT" 2020-11-19T03:31:28.632 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:28.632 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2020-11-19T03:31:28.633 INFO:teuthology.orchestra.run.smithi066.stdout: "swift_keys": [], 2020-11-19T03:31:28.633 INFO:teuthology.orchestra.run.smithi066.stdout: "caps": [], 2020-11-19T03:31:28.633 INFO:teuthology.orchestra.run.smithi066.stdout: "op_mask": "read, write, delete", 2020-11-19T03:31:28.633 INFO:teuthology.orchestra.run.smithi066.stdout: "default_placement": "", 2020-11-19T03:31:28.633 INFO:teuthology.orchestra.run.smithi066.stdout: "default_storage_class": "", 2020-11-19T03:31:28.634 INFO:teuthology.orchestra.run.smithi066.stdout: "placement_tags": [], 2020-11-19T03:31:28.634 INFO:teuthology.orchestra.run.smithi066.stdout: "bucket_quota": { 2020-11-19T03:31:28.634 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:31:28.634 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:31:28.634 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:31:28.635 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:31:28.635 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:31:28.635 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:28.635 INFO:teuthology.orchestra.run.smithi066.stdout: "user_quota": { 2020-11-19T03:31:28.636 INFO:teuthology.orchestra.run.smithi066.stdout: "enabled": false, 2020-11-19T03:31:28.636 INFO:teuthology.orchestra.run.smithi066.stdout: "check_on_raw": false, 2020-11-19T03:31:28.636 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size": -1, 2020-11-19T03:31:28.636 INFO:teuthology.orchestra.run.smithi066.stdout: "max_size_kb": 0, 2020-11-19T03:31:28.636 INFO:teuthology.orchestra.run.smithi066.stdout: "max_objects": -1 2020-11-19T03:31:28.637 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:28.637 INFO:teuthology.orchestra.run.smithi066.stdout: "temp_url_keys": [], 2020-11-19T03:31:28.637 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "rgw", 2020-11-19T03:31:28.637 INFO:teuthology.orchestra.run.smithi066.stdout: "mfa_ids": [] 2020-11-19T03:31:28.637 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:31:28.638 INFO:teuthology.orchestra.run.smithi066.stdout: 2020-11-19T03:31:28.666 INFO:rgw_multi.tests:meta checkpoint 2020-11-19T03:31:28.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:28.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:28.668 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:31:28.710 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:28.710 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:28.933 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:31:28.970 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:31:28.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:28.970 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:31:28.971 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:31:28.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:28.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:28.971 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:31:28.971 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:31:28.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:28.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:28.972 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:31:28.972 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:31:28.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:31:28.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:31:28.973 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:31:28.973 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:31:28.973 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:31:28.974 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:31:28.975 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:31:28.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:28.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:28.976 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:31:29.022 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:29.022 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:29.287 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:29.288 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:31:29.288 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:31:29.288 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:31:29.288 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:31:29.288 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:29.289 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:29.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:29.289 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:31:29.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:29.290 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:31:29.290 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:29.290 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:29.290 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:29.290 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:29.291 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:31:29.291 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:29.291 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:29.291 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:29.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:29.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:29.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:29.293 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:31:29.293 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:29.293 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:29.294 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:29.294 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:29.294 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:29.294 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:29.294 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:29.295 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:29.295 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:29.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:29.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:29.295 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:31:29.296 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:29.296 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:29.296 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:29.296 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:29.296 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:29.297 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:29.297 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:29.297 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:29.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:29.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:29.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:29.298 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:31:29.298 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:29.298 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:29.299 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:29.299 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:29.299 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:29.299 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:29.299 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:29.300 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:29.300 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:29.300 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:29.300 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:29.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:29.301 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:31:29.301 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:31:29.301 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:31:29.301 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:29.301 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:29.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:31:34.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:34.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:34.305 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:31:34.350 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:34.351 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:34.603 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:34.603 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:31:34.603 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:31:34.604 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:31:34.604 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:31:34.604 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:34.604 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:34.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:34.605 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:31:34.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:34.606 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:31:34.606 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:34.606 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:34.606 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:34.606 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:34.607 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:31:34.607 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:34.607 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:34.607 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:34.608 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:34.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:34.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:34.608 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:31:34.609 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:34.609 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:34.609 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:34.609 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:34.610 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:34.610 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:34.610 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:34.610 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:34.611 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:34.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:34.611 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:34.612 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:31:34.612 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:34.612 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:34.612 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:34.613 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:34.613 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:34.613 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:34.614 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:34.614 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:34.614 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:34.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:34.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:34.615 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:31:34.616 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:34.616 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:34.616 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:34.616 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:34.617 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:34.617 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:34.617 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:34.618 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:34.618 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:34.618 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:34.618 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:34.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:34.619 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:31:34.619 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:31:34.620 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:31:34.620 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:34.620 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:34.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:31:39.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:39.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:39.684 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:31:39.727 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:39.728 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:39.986 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:39.986 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:31:39.987 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:31:39.987 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:31:39.987 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:31:39.987 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:39.988 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:39.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:39.988 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:31:39.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:39.988 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:31:39.989 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:39.989 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:39.989 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:39.989 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:39.990 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:31:39.990 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:39.990 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:39.990 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:39.990 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:39.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:39.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:39.991 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:31:39.991 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:39.991 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:39.992 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:39.992 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:39.992 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:39.992 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:39.993 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:39.993 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:39.993 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:39.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:39.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:39.994 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:31:39.994 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:39.994 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:39.995 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:39.995 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:39.995 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:39.995 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:39.995 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:39.996 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:39.996 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:39.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:39.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:39.997 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:31:39.997 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:39.997 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:39.997 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:39.997 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:39.998 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:39.998 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:39.998 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:39.998 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:39.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:39.999 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:39.999 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:39.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:39.999 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:31:39.999 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:31:40.000 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:31:40.000 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:40.000 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:40.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:31:45.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:45.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:45.699 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:31:45.746 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:45.746 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:45.994 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:45.994 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:31:45.994 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:31:45.995 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:31:45.995 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:31:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:45.996 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:31:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:31:45.997 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:45.998 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:45.998 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:45.998 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:45.999 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:31:45.999 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:45.999 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:46.000 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:46.000 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:46.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:46.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:46.001 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:31:46.002 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:46.002 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:46.003 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:46.003 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:46.003 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:46.004 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:46.004 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:46.005 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:46.005 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:46.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:46.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:46.006 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:31:46.006 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:46.007 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:46.007 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:46.007 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:46.008 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:46.008 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:46.008 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:46.009 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:46.009 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:46.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:46.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:46.010 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:31:46.010 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:46.010 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:46.011 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:46.011 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:46.011 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:46.012 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:46.013 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:46.013 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:46.013 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:46.014 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:46.014 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:46.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:46.015 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:31:46.015 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:31:46.015 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:31:46.016 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:46.016 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:46.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:31:51.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:51.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:51.020 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:31:51.062 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:51.063 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:51.326 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:51.327 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:31:51.327 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:31:51.328 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:31:51.328 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:31:51.328 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:51.328 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:51.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:51.329 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:31:51.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:51.331 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:31:51.333 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:51.333 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:51.334 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:51.334 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:51.334 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:31:51.335 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:51.335 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:51.335 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:51.335 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:51.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:51.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:51.336 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:31:51.337 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:51.337 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:51.337 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:51.337 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:51.338 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:51.338 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:51.338 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:51.339 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:51.339 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:51.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:51.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:51.340 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:31:51.340 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:51.340 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:51.341 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:51.341 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:51.341 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:51.341 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:51.342 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:51.342 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:51.342 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:51.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:51.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:51.343 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:31:51.343 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:51.344 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:51.344 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:51.344 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:51.344 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:51.345 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:51.345 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:51.345 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:51.346 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:51.346 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:51.346 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:51.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:51.347 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:31:51.347 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:31:51.348 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:31:51.348 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:51.349 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:51.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:31:56.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:56.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:31:56.352 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:31:56.394 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:31:56.395 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:31:56.639 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:31:56.640 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:31:56.640 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:31:56.640 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:31:56.641 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:31:56.641 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:31:56.641 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:56.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:56.641 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:31:56.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:56.642 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:31:56.642 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:56.642 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:56.643 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:56.643 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:56.643 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:31:56.643 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:56.643 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:56.644 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:56.644 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:56.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:56.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:56.644 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:31:56.645 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:56.645 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:56.645 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:56.645 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:56.645 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:56.646 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:56.646 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:56.646 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:56.646 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:56.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:56.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:56.647 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:31:56.647 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:56.647 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:56.648 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:56.648 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:56.648 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:56.648 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:56.648 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:56.649 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:56.649 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:56.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:56.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:31:56.650 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:31:56.650 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:31:56.650 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:31:56.650 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:31:56.650 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:31:56.651 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:31:56.651 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:31:56.651 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:31:56.651 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:31:56.651 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:56.652 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:56.652 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:31:56.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:31:56.652 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:31:56.652 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:31:56.653 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:31:56.653 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:31:56.653 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:31:56.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:01.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:01.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:01.658 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:01.699 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:01.699 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:01.937 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:01.938 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:01.938 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:01.938 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:01.938 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:01.939 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:01.939 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:01.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:01.939 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:01.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:01.940 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:01.940 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:01.940 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:01.940 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:01.940 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:01.941 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:01.941 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:01.941 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:01.941 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:01.941 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:01.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:01.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:01.942 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:01.942 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:01.942 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:01.943 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:01.943 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:01.943 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:01.943 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:01.943 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:01.944 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:01.944 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:01.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:01.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:01.944 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:01.945 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:01.945 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:01.945 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:01.945 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:01.945 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:01.946 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:01.946 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:01.946 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:01.946 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:01.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:01.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:01.947 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:01.947 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:01.947 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:01.947 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:01.948 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:01.948 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:01.948 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:01.948 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:01.948 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:01.949 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:01.949 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:01.949 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:01.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:01.949 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:01.950 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:01.950 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:01.950 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:01.950 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:01.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:06.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:06.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:06.954 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:06.996 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:06.996 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:07.250 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:07.250 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:07.251 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:07.251 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:07.251 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:07.251 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:07.252 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:07.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:07.252 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:07.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:07.253 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:07.253 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:07.253 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:07.253 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:07.254 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:07.254 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:07.254 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:07.254 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:07.254 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:07.255 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:07.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:07.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:07.255 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:07.255 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:07.256 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:07.256 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:07.256 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:07.256 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:07.257 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:07.257 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:07.257 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:07.257 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:07.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:07.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:07.258 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:07.258 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:07.258 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:07.259 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:07.259 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:07.259 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:07.259 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:07.259 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:07.260 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:07.260 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:07.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:07.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:07.260 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:07.261 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:07.261 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:07.261 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:07.261 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:07.261 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:07.262 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:07.262 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:07.262 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:07.262 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:07.262 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:07.263 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:07.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:07.263 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:07.263 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:07.264 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:07.264 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:07.264 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:07.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:12.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:12.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:12.268 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:12.327 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:12.327 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:12.572 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:12.572 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:12.572 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:12.572 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:12.573 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:12.573 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:12.573 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:12.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:12.574 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:12.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:12.574 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:12.574 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:12.574 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:12.575 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:12.575 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:12.575 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:12.575 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:12.575 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:12.576 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:12.576 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:12.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:12.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:12.576 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:12.577 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:12.577 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:12.577 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:12.577 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:12.578 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:12.578 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:12.578 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:12.578 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:12.578 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:12.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:12.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:12.579 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:12.579 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:12.580 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:12.580 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:12.580 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:12.580 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:12.580 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:12.581 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:12.581 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:12.581 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:12.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:12.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:12.582 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:12.582 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:12.582 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:12.582 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:12.583 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:12.583 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:12.583 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:12.583 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:12.584 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:12.584 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:12.584 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:12.584 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:12.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:12.585 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:12.585 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:12.585 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:12.585 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:12.585 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:12.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:17.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:17.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:17.588 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:17.631 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:17.631 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:17.868 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:17.882 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:17.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:22.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:22.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:22.885 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:22.930 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:22.931 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:23.182 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:23.182 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:23.182 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:23.183 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:23.183 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:23.183 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:23.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:23.183 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:23.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:23.184 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:23.184 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:23.184 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:23.184 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:23.185 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:23.185 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:23.185 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:23.185 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:23.186 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:23.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:23.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:23.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:23.186 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:23.187 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:23.187 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:23.187 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:23.187 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:23.187 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:23.188 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:23.188 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:23.188 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:23.188 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:23.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:23.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:23.189 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:23.189 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:23.189 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:23.190 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:23.190 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:23.190 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:23.190 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:23.190 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:23.191 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:23.191 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:23.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:23.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:23.191 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:23.192 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:23.192 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:23.192 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:23.192 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:23.192 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:23.193 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:23.193 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:23.193 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:23.193 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:23.193 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:23.194 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:23.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:23.194 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:23.194 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:23.195 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:23.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:23.195 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:23.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:28.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:28.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:28.198 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:28.242 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:28.242 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:28.509 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:28.509 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:28.510 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:28.510 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:28.511 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:28.511 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:28.511 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:28.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:28.512 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:28.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:28.513 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:28.513 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:28.514 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:28.514 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:28.514 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:28.515 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:28.515 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:28.516 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:28.516 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:28.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:28.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:28.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:28.517 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:28.518 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:28.518 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:28.518 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:28.519 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:28.519 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:28.519 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:28.520 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:28.520 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:28.520 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:28.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:28.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:28.528 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:28.528 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:28.528 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:28.528 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:28.529 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:28.529 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:28.529 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:28.529 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:28.529 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:28.530 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:28.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:28.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:28.530 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:28.531 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:28.531 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:28.531 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:28.531 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:28.531 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:28.532 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:28.532 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:28.532 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:28.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:28.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:28.533 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:28.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:28.533 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:28.533 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:28.533 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:28.534 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:28.534 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:28.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:33.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:33.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:33.536 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:33.589 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:33.589 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:33.831 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:33.831 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:33.831 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:33.832 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:33.832 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:33.832 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:33.832 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:33.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:33.833 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:33.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:33.833 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:33.833 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:33.834 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:33.834 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:33.834 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:33.834 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:33.835 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:33.846 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:33.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:38.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:38.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:38.849 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:38.890 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:38.890 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:39.156 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:39.194 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:39.194 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:39.194 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:39.194 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:39.195 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:39.195 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:39.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:39.195 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:39.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:39.196 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:39.196 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:39.196 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:39.196 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:39.197 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:39.197 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:39.197 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:39.197 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:39.197 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:39.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:39.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:39.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:39.198 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:39.198 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:39.199 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:39.199 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:39.199 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:39.199 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:39.199 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:39.200 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:39.200 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:39.200 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:39.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:39.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:39.201 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:39.201 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:39.201 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:39.201 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:39.202 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:39.202 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:39.202 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:39.202 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:39.202 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:39.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:39.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:39.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:39.203 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:39.203 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:39.204 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:39.204 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:39.204 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:39.204 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:39.204 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:39.205 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:39.205 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:39.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:39.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:39.206 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:39.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:39.206 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:39.206 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:39.207 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:39.207 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:39.207 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:39.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:44.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:45.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:45.347 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:45.390 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:45.390 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:45.644 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:45.644 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:45.644 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:45.645 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:45.645 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:45.645 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:45.646 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:45.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:45.646 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:45.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:45.646 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:45.647 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:45.647 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:45.647 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:45.647 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:45.648 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:45.648 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:45.648 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:45.648 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:45.649 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:45.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:45.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:45.649 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:45.649 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:45.650 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:45.650 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:45.650 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:45.650 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:45.650 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:45.651 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:45.651 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:45.651 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:45.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:45.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:45.652 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:45.652 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:45.652 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:45.652 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:45.653 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:45.653 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:45.653 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:45.654 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:45.654 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:45.654 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:45.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:45.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:45.655 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:45.655 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:45.655 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:45.655 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:45.656 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:45.656 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:45.656 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:45.656 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:45.656 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:45.657 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:45.657 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:45.657 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:45.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:45.658 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:45.658 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:45.658 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:45.658 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:45.658 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:45.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:50.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:51.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:51.074 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:51.115 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:51.116 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:51.512 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:51.512 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:51.512 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:51.512 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:51.513 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:51.513 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:51.513 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:51.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:51.514 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:51.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:51.514 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:51.514 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:51.514 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:51.515 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:51.515 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:51.515 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:51.515 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:51.515 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:51.516 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:51.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:51.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:51.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:51.516 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:51.517 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:51.517 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:51.517 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:51.517 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:51.517 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:51.518 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:51.518 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:51.518 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:51.518 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:51.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:51.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:51.519 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:51.519 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:51.519 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:51.519 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:51.520 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:51.520 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:51.520 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:51.520 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:51.520 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:51.521 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:51.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:51.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:51.521 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:51.522 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:51.522 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:51.522 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:51.522 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:51.522 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:51.523 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:51.523 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:51.523 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:51.523 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:51.523 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:51.524 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:51.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:51.524 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:51.524 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:51.524 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:51.525 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:51.525 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:51.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:32:56.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:56.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:32:56.528 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:32:56.568 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:32:56.568 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:32:56.810 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:32:56.810 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:32:56.811 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:32:56.811 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:32:56.811 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:32:56.811 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:32:56.812 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:56.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:56.812 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:32:56.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:56.812 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:32:56.813 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:56.813 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:56.813 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:56.813 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:56.814 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:32:56.814 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:56.814 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:56.814 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:56.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:56.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:56.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:56.815 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:32:56.815 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:56.815 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:56.816 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:56.816 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:56.816 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:56.816 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:56.817 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:56.817 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:56.817 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:56.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:56.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:56.818 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:32:56.818 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:56.818 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:56.818 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:56.819 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:56.819 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:56.819 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:56.819 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:56.819 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:56.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:56.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:56.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:32:56.820 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:32:56.821 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:32:56.821 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:32:56.821 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:32:56.821 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:32:56.822 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:32:56.822 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:32:56.822 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:32:56.822 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:32:56.822 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:56.823 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:56.823 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:32:56.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:32:56.823 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:32:56.823 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:32:56.824 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:32:56.824 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:32:56.824 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:32:56.825 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:33:01.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:01.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:01.828 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:01.889 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:01.891 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:02.138 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:02.138 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:02.139 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:02.139 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:02.140 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:02.140 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:02.141 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:02.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:02.141 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:02.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:02.142 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:02.143 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:02.143 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:02.143 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:02.144 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:02.144 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:02.144 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:02.145 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:02.145 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:02.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:02.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:02.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:02.147 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:02.147 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:02.148 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:02.148 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:02.148 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:02.149 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:02.149 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:02.149 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:02.150 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:02.150 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:02.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:02.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:02.151 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:02.151 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:02.151 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:02.151 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:02.152 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:02.152 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:02.152 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:02.152 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:02.152 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:02.153 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:02.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:02.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:02.153 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:02.153 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:02.154 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:02.154 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:02.154 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:02.154 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:02.154 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:02.155 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:02.155 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:02.155 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:02.155 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:02.156 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:02.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:02.156 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:02.156 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:02.156 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:02.157 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:02.157 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:02.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:33:07.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:07.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:07.354 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:07.397 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:07.397 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:07.663 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:07.663 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:07.663 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:07.664 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:07.664 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:07.664 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:07.665 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:07.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:07.665 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:07.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:07.666 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:07.666 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:07.666 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:07.666 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:07.667 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:07.667 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:07.667 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:07.667 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:07.668 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:07.668 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:07.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:07.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:07.669 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:07.669 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:07.669 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:07.669 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:07.670 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:07.670 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:07.670 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:07.670 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:07.671 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:07.671 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:07.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:07.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:07.671 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:07.672 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:07.672 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:07.672 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:07.672 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:07.673 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:07.673 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:07.673 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:07.674 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:07.674 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:07.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:07.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:07.675 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:07.675 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:07.675 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:07.675 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:07.675 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:07.676 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:07.676 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:07.676 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:07.677 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:07.677 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:07.677 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:07.677 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:07.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:07.678 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:07.678 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:07.678 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:07.679 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:07.679 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:07.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:33:12.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:12.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:12.683 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:12.718 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:12.718 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:12.964 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:12.993 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:12.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:33:17.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:17.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:17.999 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:18.051 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:18.052 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:18.327 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:18.327 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:18.327 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:18.341 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:18.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1605756688.618934_42.1 target= 2020-11-19T03:33:23.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:23.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:23.344 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:23.389 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:23.389 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:23.660 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:23.660 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:23.661 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:23.661 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:23.661 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:23.661 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:23.661 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:23.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:23.662 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:23.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:23.662 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:23.662 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:23.663 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:23.663 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:23.663 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:23.663 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:23.663 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:23.664 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:23.664 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:23.664 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:23.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:23.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:23.665 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:23.665 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:23.665 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:23.665 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:23.667 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:23.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:23.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:23.667 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:23.667 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:23.668 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:23.668 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:23.668 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:23.668 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:23.668 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:23.669 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:23.669 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:23.669 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:23.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:23.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:23.670 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:23.670 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:23.670 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:23.670 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:23.670 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:23.671 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:23.671 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:23.671 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:23.671 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:23.672 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:23.672 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:23.672 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:23.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:23.672 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:23.673 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:23.673 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:23.673 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:23.673 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:23.674 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:33:23.674 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:33:23.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:23.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:23.675 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:23.720 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:23.721 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:23.995 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:33:23.996 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:33:23.996 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:33:23.996 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:33:23.997 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:33:23.997 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:23.997 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:23.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:23.998 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:33:23.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:23.998 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:33:23.998 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:23.998 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:23.999 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:23.999 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:23.999 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:33:23.999 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:23.999 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:24.000 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:24.000 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:24.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:24.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:24.001 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:33:24.001 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:24.001 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:24.001 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:24.001 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:24.002 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:24.002 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:24.002 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.002 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:24.002 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:24.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:24.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:24.003 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:33:24.003 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:24.003 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:24.004 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:24.004 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:24.004 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:24.004 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:24.005 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.005 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:24.005 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:24.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:24.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:24.006 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:33:24.006 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:24.006 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:24.006 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:24.006 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:24.007 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:24.007 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:24.007 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.007 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:24.007 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:24.008 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:24.008 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:33:24.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:24.008 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:33:24.009 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:33:24.009 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:33:24.009 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:24.009 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:33:24.010 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:33:24.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:33:24.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.012 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:24.064 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:24.065 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:24.333 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:24.333 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:24.333 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:24.334 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:24.334 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:24.334 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:24.334 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.335 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:24.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.335 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:24.335 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.335 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.336 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:24.336 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.336 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:24.336 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.337 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:24.337 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.337 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.338 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:24.338 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.338 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.338 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:24.338 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.339 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:24.339 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.339 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.339 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.339 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.340 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:24.340 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.340 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.341 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:24.341 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.341 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:24.341 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.341 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.342 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.342 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.342 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:24.343 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.343 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.343 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:24.343 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.344 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:24.344 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.344 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.344 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.344 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.345 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.345 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:24.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.345 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:24.346 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:24.346 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:24.346 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.346 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:24.347 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:33:24.348 INFO:rgw_multi.tests:meta checkpoint 2020-11-19T03:33:24.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.349 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:24.393 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:24.393 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:24.637 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:33:24.670 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:33:24.671 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:24.671 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:24.671 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:33:24.672 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:24.672 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:24.672 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:24.672 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:24.672 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:24.673 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:24.673 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:24.673 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:24.673 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:24.673 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:24.674 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:24.674 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:24.674 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:24.674 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:33:24.675 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:33:24.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.677 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:24.724 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:24.724 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:24.965 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:24.966 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:24.967 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:24.968 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:24.969 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:24.969 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:24.970 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.972 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:24.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.973 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:24.973 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.973 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.973 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:24.974 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.974 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:24.974 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.974 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:24.975 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.975 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.975 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:24.976 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.976 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.976 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:24.976 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.976 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:24.977 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.977 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.977 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.977 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.978 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:24.978 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.978 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.979 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:24.979 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.979 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:24.979 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.979 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.980 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.980 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:24.980 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:24.981 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:24.981 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:24.981 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:24.981 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:24.981 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:24.982 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:24.982 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:24.982 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:24.982 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.983 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.983 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:24.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:24.983 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:24.983 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:24.984 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:24.984 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:24.984 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:24.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:33:24.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:33:24.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:24.987 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:25.030 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:25.031 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:25.279 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:33:25.280 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:33:25.280 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:33:25.280 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:33:25.281 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:33:25.281 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:25.281 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:25.281 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:25.281 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:33:25.282 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:25.282 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:33:25.282 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:25.282 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:25.282 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:25.283 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:25.283 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:33:25.283 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:25.283 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:25.283 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:25.284 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:25.284 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:25.284 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:25.284 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:33:25.284 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:25.285 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:25.285 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:25.285 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:25.285 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:25.286 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:25.286 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:25.286 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:25.286 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:25.286 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:25.287 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:25.287 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:33:25.287 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:25.287 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:25.287 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:25.288 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:25.288 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:25.288 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:25.288 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:25.288 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:25.289 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:25.289 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:25.289 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:25.289 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:33:25.289 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:25.290 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:25.290 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:25.290 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:25.290 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:25.290 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:25.291 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:25.291 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:25.291 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:25.291 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:25.291 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:33:25.292 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:25.292 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:33:25.292 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:33:25.292 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:33:25.292 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:25.293 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:33:25.294 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:33:25.294 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:33:25.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:25.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:25.295 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:25.340 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:25.340 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:25.599 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:25.600 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:25.600 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:25.600 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:25.601 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:25.601 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:25.601 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:25.601 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:25.601 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:25.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:25.602 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:25.602 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:25.602 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:25.603 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:25.603 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:25.603 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:25.603 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:25.604 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:25.604 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:25.604 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:25.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:25.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:25.605 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:25.605 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:25.605 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:25.606 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:25.606 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:25.606 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:25.606 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:25.606 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:25.607 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:25.607 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:25.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:25.607 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:25.607 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:25.608 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:25.608 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:25.608 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:25.608 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:25.608 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:25.609 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:25.609 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:25.609 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:25.609 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:25.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:25.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:25.610 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:25.610 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:25.610 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:25.611 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:25.611 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:25.611 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:25.611 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:25.611 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:25.612 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:25.612 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:25.612 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:25.612 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:25.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:25.613 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:25.613 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:25.613 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:25.613 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:25.613 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:25.614 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:33:25.615 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-11-19T03:33:25.623 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kmbxto-1 2020-11-19T03:33:29.261 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kmbxto-2 2020-11-19T03:33:32.837 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kmbxto-3 2020-11-19T03:33:32.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:32.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:32.910 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:32.954 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:32.955 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:33.168 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:33:33.202 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:33:33.202 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.203 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:33.203 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:33:33.203 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.203 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.203 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:33.204 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:33.204 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.204 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.204 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756812.884086_172.1", 2020-11-19T03:33:33.205 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:32.884086Z" 2020-11-19T03:33:33.205 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.205 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.205 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:33.205 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:33.206 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:33.206 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:33:33.207 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:33:33.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:33.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:33.210 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:33.251 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:33.252 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:33.519 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:33.520 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:33.520 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:33.520 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:33.520 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:33.521 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:33.521 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:33.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:33.521 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:33.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:33.522 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:33.522 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:33.522 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:33.522 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:33.522 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:33.523 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:33.523 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:33.523 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:33.523 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:33.524 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:33.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:33.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:33.524 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:33.524 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:33.525 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:33.525 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:33.525 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:33.525 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:33.525 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:33.526 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:33.527 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:33.527 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:33.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:33.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:33.527 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:33.528 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:33.528 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:33.528 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756812.884086_172.1", 2020-11-19T03:33:33.528 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:33.529 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:33.529 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:33.529 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:32.884086Z", 2020-11-19T03:33:33.529 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:33.530 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:33.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:33.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:33.530 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:33.530 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:33.531 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:33.531 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:33.531 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:33.531 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:33.532 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:33.532 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:33.532 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:33.533 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:33.533 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:33.533 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:33.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:33.534 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:33.534 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:33.534 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:33.534 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:33.534 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:33.535 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:33:33.536 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:33:33.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:33.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:33.538 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:33.587 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:33.588 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:33.834 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:33:33.834 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:33:33.834 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:33:33.835 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:33:33.835 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:33:33.835 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:33.835 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:33.835 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.836 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:33:33.836 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.836 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:33:33.836 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:33.837 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:33.837 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:33.837 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:33.837 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:33:33.838 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:33.838 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:33.838 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:33.838 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:33.838 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.839 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.839 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:33:33.839 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:33.839 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:33.839 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:33.840 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:33.840 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:33.840 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:33.840 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:33.841 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:33.841 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:33.841 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.841 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.841 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:33:33.842 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:33.842 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:33.842 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756812.884086_172.1", 2020-11-19T03:33:33.842 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:33.842 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:33.843 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:33.843 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:32.884086Z", 2020-11-19T03:33:33.844 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:33.844 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:33.844 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.844 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:33.845 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:33:33.845 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:33.845 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:33.845 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:33.846 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:33.846 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:33.846 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:33.847 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:33.847 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:33.847 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:33.847 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:33.847 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:33:33.848 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:33.848 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:33:33.848 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:33:33.848 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:33:33.848 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:33.849 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:33:33.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:33:33.850 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:33:33.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:33.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:33.852 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:33.895 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:33.895 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:34.164 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:34.502 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:34.503 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:34.503 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:34.503 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:34.504 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:34.504 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:34.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:34.505 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:34.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:34.505 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:34.505 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:34.506 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:34.511 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:34.511 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:34.511 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:34.512 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:34.512 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:34.512 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:34.512 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:34.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:34.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:34.513 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:34.513 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:34.513 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:34.514 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:34.514 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:34.514 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:34.514 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:34.514 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:34.515 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:34.515 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:34.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:34.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:34.515 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:34.516 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:34.516 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:34.516 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756812.884086_172.1", 2020-11-19T03:33:34.516 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:34.517 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:34.517 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:34.517 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:32.884086Z", 2020-11-19T03:33:34.518 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:34.518 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:34.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:34.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:34.518 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:34.519 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:34.519 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:34.519 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:34.519 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:34.519 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:34.520 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:34.520 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:34.520 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:34.520 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:34.521 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:34.521 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:34.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:34.521 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:34.521 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:34.522 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:34.522 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:34.522 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:34.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:33:37.231 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-11-19T03:33:37.234 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kmbxto-4 2020-11-19T03:33:38.024 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kmbxto-5 2020-11-19T03:33:38.664 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kmbxto-6 2020-11-19T03:33:39.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:39.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:39.791 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:39.834 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:39.834 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:40.062 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:33:40.102 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:33:40.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.103 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:40.103 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:33:40.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.104 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.105 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:40.105 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:40.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.106 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.106 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:40.107 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:39.214864Z" 2020-11-19T03:33:40.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.108 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.109 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:40.109 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:40.110 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:40.110 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:33:40.112 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:33:40.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:40.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:40.115 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:40.161 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:40.162 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:40.406 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:40.406 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:40.407 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:40.407 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:40.407 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:40.408 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:40.408 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:40.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:40.409 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:40.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:40.410 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:40.411 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:40.411 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:40.411 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:40.411 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:40.412 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:40.412 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:40.412 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:40.412 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:40.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:40.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:40.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:40.413 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:40.413 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:40.414 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:40.414 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:40.414 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:40.414 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:40.415 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:40.415 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:40.415 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:40.415 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:40.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:40.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:40.416 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:40.416 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:40.418 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:40.418 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:40.419 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:40.419 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:40.419 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:40.420 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:39.214864Z", 2020-11-19T03:33:40.421 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:40.421 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:40.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:40.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:40.422 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:40.423 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:40.423 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:40.423 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:40.424 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:40.424 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:40.424 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:40.425 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:40.425 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:40.426 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:40.426 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:40.427 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:40.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:40.428 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:40.429 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:40.429 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:40.429 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:40.430 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:40.430 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:33:40.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:33:40.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:40.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:40.432 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:40.473 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:40.473 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:40.751 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:33:40.751 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:33:40.752 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:33:40.752 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:33:40.752 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:33:40.752 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:40.752 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:40.753 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.753 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:33:40.753 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.753 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:33:40.754 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:40.754 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:40.754 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:40.754 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:40.754 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:33:40.755 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:40.755 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:40.755 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:40.755 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:40.755 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.756 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.756 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:33:40.756 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:40.756 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:40.756 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:40.757 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:40.757 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:40.757 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:40.757 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:40.760 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:40.760 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:40.761 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.761 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.761 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:33:40.762 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:40.762 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:40.763 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:40.763 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:40.763 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:40.764 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:40.764 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:39.214864Z", 2020-11-19T03:33:40.764 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:40.765 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:40.765 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.765 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:40.766 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:33:40.766 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:40.766 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:40.766 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:40.767 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:40.767 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:40.767 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:40.768 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:40.768 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:40.768 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:40.769 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:40.769 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:33:40.769 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:40.770 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:33:40.770 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:33:40.770 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:33:40.771 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:40.771 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:33:40.773 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:33:40.774 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:33:40.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:40.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:40.777 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:40.818 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:40.819 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:41.086 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:41.087 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:41.087 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:41.087 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:41.088 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:41.088 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:41.088 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:41.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:41.089 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:41.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:41.089 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:41.089 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:41.090 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:41.090 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:41.090 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:41.090 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:41.091 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:41.091 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:41.091 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:41.091 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:41.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:41.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:41.092 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:41.092 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:41.092 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:41.093 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:41.093 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:41.093 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:41.093 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:41.094 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:41.094 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:41.094 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:41.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:41.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:41.095 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:41.095 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:41.095 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:41.096 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:41.096 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:41.096 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:41.096 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:41.097 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:39.214864Z", 2020-11-19T03:33:41.097 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:41.097 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:41.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:41.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:41.098 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:41.098 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:41.098 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:41.099 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:41.099 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:41.099 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:41.099 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:41.100 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:41.100 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:41.100 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:41.100 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:41.101 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:41.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:41.101 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:41.101 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:41.102 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:41.102 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:41.102 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:41.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:33:42.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:42.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:42.210 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:42.252 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:42.253 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:42.472 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:33:42.503 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:33:42.503 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:42.504 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:42.504 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:33:42.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:42.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:42.504 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:42.505 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:42.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:42.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:42.505 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:42.505 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:39.214864Z" 2020-11-19T03:33:42.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:42.506 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:42.506 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:42.506 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:42.506 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:42.507 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:33:42.508 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:33:42.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:42.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:42.509 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:42.553 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:42.553 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:42.795 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:42.795 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:42.796 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:42.796 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:42.796 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:42.796 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:42.796 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:42.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:42.797 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:42.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:42.797 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:42.797 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:42.798 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:42.798 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:42.798 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:42.798 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:42.798 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:42.799 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:42.799 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:42.799 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:42.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:42.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:42.800 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:42.800 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:42.800 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:42.800 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:42.801 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:42.801 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:42.801 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:42.801 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:42.801 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:42.802 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:42.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:42.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:42.802 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:42.802 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:42.803 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:42.803 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:42.803 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:42.803 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:42.803 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:42.804 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:39.214864Z", 2020-11-19T03:33:42.804 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:42.804 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:42.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:42.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:42.805 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:42.805 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:42.805 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:42.805 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:42.805 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:42.806 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:42.806 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:42.806 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:42.806 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:42.806 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:42.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:42.807 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:42.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:42.807 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:42.807 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:42.808 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:42.808 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:42.808 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:42.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:33:42.810 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:33:42.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:42.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:42.811 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:42.854 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:42.855 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:43.110 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:33:43.111 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:33:43.111 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:33:43.111 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:33:43.111 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:33:43.112 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:43.112 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:43.112 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:43.112 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:33:43.112 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:43.113 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:33:43.113 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:43.113 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:43.113 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:43.114 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:43.114 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:33:43.114 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:43.114 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:43.114 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:43.115 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:43.115 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:43.115 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:43.115 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:33:43.115 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:43.116 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:43.116 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:43.116 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:43.116 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:43.116 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:43.117 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:43.117 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:43.117 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:43.117 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:43.117 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:43.118 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:33:43.118 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:43.118 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:43.118 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:43.118 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:43.119 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:43.119 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:43.119 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:39.214864Z", 2020-11-19T03:33:43.119 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:43.120 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:43.120 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:43.120 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:43.120 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:33:43.120 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:43.121 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:43.121 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:43.121 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:43.121 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:43.121 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:43.122 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:43.122 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:43.122 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:43.122 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:43.122 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:33:43.123 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:43.123 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:33:43.123 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:33:43.123 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:33:43.123 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:43.124 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:33:43.125 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:33:43.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:33:43.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:43.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:43.127 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:43.167 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:43.168 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:43.436 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756819.214864_185.1", 2020-11-19T03:33:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:39.214864Z", 2020-11-19T03:33:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:43.457 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:43.458 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:33:45.817 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-11-19T03:33:45.818 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kmbxto-7 2020-11-19T03:33:46.133 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kmbxto-8 2020-11-19T03:33:46.887 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kmbxto-9 2020-11-19T03:33:47.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:47.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:47.354 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:47.392 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:47.392 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:47.627 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:33:47.660 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:33:47.660 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:47.660 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:47.660 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:33:47.661 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:47.661 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:47.661 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:47.661 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:47.662 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:47.662 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:47.662 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756827.330725_199.1", 2020-11-19T03:33:47.662 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:47.330725Z" 2020-11-19T03:33:47.662 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:47.663 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:47.663 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:47.663 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:47.664 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:47.664 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:33:47.666 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:33:47.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:47.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:47.669 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:47.718 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:47.718 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:47.962 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:47.962 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:47.963 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:47.963 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:47.963 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:47.963 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:47.963 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:47.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:47.964 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:47.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:47.964 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:47.965 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:47.965 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:47.965 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:47.966 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:47.966 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:47.966 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:47.966 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:47.967 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:47.967 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:47.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:47.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:47.967 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:47.968 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:47.968 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:47.968 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:47.968 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:47.968 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:47.969 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:47.969 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:47.969 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:47.969 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:47.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:47.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:47.970 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:47.970 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:47.970 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:47.970 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756827.330725_199.1", 2020-11-19T03:33:47.971 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:47.971 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:47.971 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:47.971 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:47.330725Z", 2020-11-19T03:33:47.971 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:47.972 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:47.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:47.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:47.972 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:47.972 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:47.973 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:47.973 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:47.973 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:47.973 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:47.973 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:47.974 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:47.974 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:47.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:47.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:47.974 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:47.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:47.975 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:47.976 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:47.976 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:47.976 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:47.976 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:47.977 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:33:47.978 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:33:47.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:47.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:47.979 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:48.020 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:48.021 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:48.278 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:33:48.278 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:33:48.278 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:33:48.279 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:33:48.279 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:33:48.279 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:48.279 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:48.279 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:48.280 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:33:48.280 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:48.280 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:33:48.280 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:48.281 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:48.281 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:48.281 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:48.281 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:33:48.281 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:48.282 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:48.282 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:48.282 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:48.283 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:48.283 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:48.283 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:33:48.284 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:48.284 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:48.284 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:48.284 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:48.284 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:48.285 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:48.285 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:48.285 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:48.285 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:48.286 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:48.286 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:48.286 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:33:48.286 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:48.286 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:48.289 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756827.330725_199.1", 2020-11-19T03:33:48.289 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:48.289 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:48.290 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:48.290 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:47.330725Z", 2020-11-19T03:33:48.290 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:48.290 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:48.291 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:48.291 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:48.291 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:33:48.291 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:48.292 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:48.292 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:48.293 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:48.293 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:48.293 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:48.293 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:48.293 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:48.294 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:48.294 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:48.294 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:33:48.295 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:48.295 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:33:48.295 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:33:48.295 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:33:48.295 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:48.296 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:33:48.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:33:48.298 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:33:48.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:48.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:48.301 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:48.345 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:48.346 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:48.616 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:48.616 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:48.616 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:48.616 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:48.617 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:48.617 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:48.617 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:48.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:48.618 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:48.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:48.619 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:48.619 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:48.620 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:48.620 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:48.621 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:48.621 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:48.621 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:48.622 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:48.622 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:48.624 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:48.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:48.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:48.625 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:48.626 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:48.626 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:48.626 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:48.627 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:48.627 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:48.628 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:48.628 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:48.628 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:48.629 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:48.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:48.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:48.630 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:48.630 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:48.631 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:48.631 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756827.330725_199.1", 2020-11-19T03:33:48.632 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:48.632 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:48.632 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:48.633 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:47.330725Z", 2020-11-19T03:33:48.633 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:48.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:48.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:48.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:48.636 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:48.636 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:48.636 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:48.637 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:48.637 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:48.638 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:48.638 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:48.638 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:48.639 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:48.639 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:48.640 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:48.640 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:48.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:48.641 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:48.641 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:48.642 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:48.642 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:48.642 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:48.643 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:33:49.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:51.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:51.182 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:51.228 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:51.229 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:51.446 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:33:51.479 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:33:51.479 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:51.479 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:51.480 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:33:51.480 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:51.480 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:51.480 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:51.481 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:51.481 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:51.481 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:51.481 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:33:51.481 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:49.439575Z" 2020-11-19T03:33:51.482 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:51.482 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:51.482 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:51.482 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:51.483 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:51.483 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:33:51.484 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:33:51.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:51.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:51.485 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:51.526 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:51.526 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:51.759 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:51.759 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:51.759 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:51.759 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:51.760 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:51.760 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:51.760 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:51.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:51.761 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:51.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:51.761 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:51.761 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:51.761 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:51.762 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:51.762 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:51.762 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:51.762 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:51.762 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:51.763 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:51.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:51.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:51.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:51.764 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:51.764 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:51.764 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:51.764 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:51.764 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:51.765 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:51.765 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:51.765 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:51.765 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:51.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:51.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:51.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:51.767 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:51.767 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:51.767 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:33:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:51.768 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:33:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:51.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:51.770 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:51.771 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:51.771 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:51.771 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:51.771 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:51.771 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:51.772 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:51.772 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:51.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:51.772 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:51.772 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:51.773 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:51.773 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:51.773 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:51.774 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:33:51.775 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:33:51.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:51.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:51.776 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:51.820 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:51.820 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:52.077 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:33:52.077 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:33:52.077 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:33:52.078 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:33:52.078 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:33:52.078 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:52.078 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:52.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:52.079 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:33:52.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:52.079 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:33:52.079 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:52.079 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:52.080 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:52.080 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:52.080 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:33:52.080 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:52.080 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:52.081 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:52.081 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:52.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:52.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:52.081 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:33:52.082 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:52.082 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:52.082 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:52.082 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:52.082 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:52.083 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:52.083 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:52.083 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:52.083 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:52.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:52.084 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:52.084 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:33:52.084 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:52.084 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:52.084 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:33:52.085 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:52.085 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:52.085 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:52.085 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:33:52.085 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:52.086 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:52.086 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:52.086 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:52.087 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:33:52.087 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:33:52.087 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:33:52.087 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:52.087 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:33:52.088 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:33:52.088 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:33:52.088 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:33:52.088 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:33:52.088 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:52.089 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:52.089 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:33:52.089 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:52.089 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:33:52.089 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:33:52.090 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:33:52.090 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:52.090 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:33:52.090 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:33:52.091 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:33:52.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:52.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:52.092 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:52.138 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:52.138 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:52.404 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:33:52.404 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:33:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:33:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:33:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:33:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:33:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:33:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:33:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:33:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:33:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:52.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:52.409 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:33:52.409 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:52.409 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:52.410 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:52.410 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:52.410 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:52.410 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:52.410 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:52.411 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:52.411 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:52.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:52.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:52.411 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:33:52.412 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:52.412 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:52.412 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:33:52.412 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:52.412 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:52.413 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:52.413 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:33:52.413 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:52.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:52.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:52.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:33:52.414 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:33:52.414 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:33:52.414 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:33:52.415 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:33:52.415 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:33:52.415 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:33:52.415 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:33:52.415 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:33:52.416 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:33:52.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:52.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:52.416 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:33:52.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:33:52.417 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:33:52.417 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:33:52.417 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:33:52.417 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:33:52.417 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:33:52.419 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:33:53.940 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-11-19T03:33:53.942 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kmbxto-10 2020-11-19T03:33:54.412 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kmbxto-11 2020-11-19T03:33:55.774 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kmbxto-12 2020-11-19T03:33:58.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:58.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:58.800 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:58.841 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:58.841 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:59.063 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:33:59.090 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:33:59.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:59.091 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:33:59.091 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:33:59.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:59.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:59.092 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:33:59.092 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:33:59.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:59.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:59.093 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:33:59.093 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:49.439575Z" 2020-11-19T03:33:59.093 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:33:59.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:33:59.094 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756836.096948_88.1", 2020-11-19T03:33:59.094 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:56.096948Z" 2020-11-19T03:33:59.094 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:33:59.094 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:33:59.095 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:33:59.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:59.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:33:59.097 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:33:59.146 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:33:59.146 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:33:59.365 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:34:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:34:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:34:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:34:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:34:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:34:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:34:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:34:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:34:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:34:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:34:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:34:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:34:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:34:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:34:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:34:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:34:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:34:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756836.096948_88.1", 2020-11-19T03:34:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:56.096948Z", 2020-11-19T03:34:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:34:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:34:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:34:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:34:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:01.508 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:34:01.509 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:34:01.510 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:34:01.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:01.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:01.511 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:34:01.554 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:01.555 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:01.831 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:34:01.832 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:34:01.832 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:34:01.832 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:34:01.832 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:34:01.833 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:34:01.833 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:01.833 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:01.833 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:34:01.833 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:01.834 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:34:01.834 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:01.834 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:01.834 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:34:01.834 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:01.835 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:34:01.835 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:01.835 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:34:01.835 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:01.836 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:01.836 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:01.836 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:01.836 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:34:01.836 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:01.837 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:01.837 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:34:01.837 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:01.837 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:34:01.837 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:01.838 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:34:01.838 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:01.838 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:01.838 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:01.838 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:01.839 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:34:01.839 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:01.839 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:01.839 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:34:01.839 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:01.840 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:34:01.840 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:01.840 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:34:01.840 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:01.840 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:01.841 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:01.841 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:01.841 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:34:01.841 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:01.841 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:01.842 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756836.096948_88.1", 2020-11-19T03:34:01.842 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:01.842 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:34:01.842 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:01.842 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:56.096948Z", 2020-11-19T03:34:01.843 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:01.843 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:01.843 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:01.843 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:34:01.843 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:01.844 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:34:01.844 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:34:01.844 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:34:01.844 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:01.844 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:34:01.846 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:34:01.846 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:34:01.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:01.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:01.847 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:34:01.896 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:01.897 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:02.166 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:34:02.166 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:34:02.167 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:34:02.167 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:34:02.167 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:34:02.167 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:34:02.168 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:02.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:02.168 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:34:02.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:02.169 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:34:02.169 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:02.169 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:02.169 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:34:02.169 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:02.170 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:34:02.170 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:02.170 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:34:02.170 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:02.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:02.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:02.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:02.171 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:34:02.171 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:02.171 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:02.172 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:34:02.172 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:02.172 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:02.172 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:02.172 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:34:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:02.174 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:34:02.174 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:02.174 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:02.174 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:34:02.174 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:02.175 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:02.175 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:02.175 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:34:02.175 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:02.175 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:02.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:02.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:02.176 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:34:02.176 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:02.176 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:02.177 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756836.096948_88.1", 2020-11-19T03:34:02.177 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:02.177 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:02.177 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:02.177 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:56.096948Z", 2020-11-19T03:34:02.178 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:02.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:02.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:02.178 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:34:02.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:02.179 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:34:02.179 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:34:02.179 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:34:02.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:02.179 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:34:02.180 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:34:02.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:02.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:02.182 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:02.272 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:02.273 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:02.507 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:02.507+0000 7fad2989b0c0 1 bucket sync caught up with source: 2020-11-19T03:34:02.507 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.6.5, 00000000001.12.5, , ] 2020-11-19T03:34:02.508 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.6.5, 00000000001.12.5, , ] 2020-11-19T03:34:02.508 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:02.507+0000 7fad2989b0c0 0 bucket checkpoint complete 2020-11-19T03:34:02.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-10 zones={test-zone2, test-zone1} 2020-11-19T03:34:04.055 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:04.197 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:04.198 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:04.198 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:04.199 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:04.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:04.209 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:04.209 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:04.210 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:04.210 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:04.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:04.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:06.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:06.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:06.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:06.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:06.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:07.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:07.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:07.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:07.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:07.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:07.678 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-10 zones={test-zone2, test-zone1} 2020-11-19T03:34:07.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:07.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:07.679 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:07.726 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:07.726 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:07.971 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:07.965+0000 7faab3aee0c0 1 bucket sync caught up with source: 2020-11-19T03:34:07.971 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.6.5, 00000000001.12.5, , ] 2020-11-19T03:34:07.971 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.6.5, 00000000001.12.5, , ] 2020-11-19T03:34:07.971 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:07.965+0000 7faab3aee0c0 0 bucket checkpoint complete 2020-11-19T03:34:08.003 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-10 zones={test-zone3, test-zone1} 2020-11-19T03:34:08.922 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:08.930 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:08.931 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:08.931 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:08.931 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:08.932 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:08.937 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:08.937 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:08.938 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:08.938 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:08.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:09.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:09.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:09.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:09.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:11.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:11.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:11.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:11.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:12.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:12.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:12.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:12.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-10 zones={test-zone3, test-zone1} 2020-11-19T03:34:12.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:12.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:12.483 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:12.533 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:12.535 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:12.746 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:12.747+0000 7fc7378d10c0 1 bucket sync caught up with source: 2020-11-19T03:34:12.746 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.6.5, 00000000001.12.5, , ] 2020-11-19T03:34:12.746 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.6.5, 00000000001.12.5, , ] 2020-11-19T03:34:12.747 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:12.747+0000 7fc7378d10c0 0 bucket checkpoint complete 2020-11-19T03:34:12.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:12.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:12.796 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:12.845 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:12.845 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:13.091 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:13.085+0000 7fb9f51bd0c0 1 bucket sync caught up with source: 2020-11-19T03:34:13.091 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.9.5, , 00000000001.9.5, , , , 00000000001.9.5, 00000000001.13.5, , ] 2020-11-19T03:34:13.091 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.9.5, , 00000000001.9.5, , , , 00000000001.9.5, 00000000001.13.5, , ] 2020-11-19T03:34:13.091 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:13.085+0000 7fb9f51bd0c0 0 bucket checkpoint complete 2020-11-19T03:34:13.122 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-11 zones={test-zone1, test-zone2} 2020-11-19T03:34:13.871 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:13.880 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:13.881 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:13.881 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:13.882 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:13.882 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:13.888 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:13.889 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:13.889 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:13.890 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:13.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:13.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:14.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:14.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:14.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:14.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:16.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:16.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:16.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:16.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:16.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:17.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:17.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-11 zones={test-zone1, test-zone2} 2020-11-19T03:34:17.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:17.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:17.146 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:17.194 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:17.195 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:17.439 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:17.433+0000 7ff8d33f00c0 1 bucket sync caught up with source: 2020-11-19T03:34:17.441 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.9.5, , 00000000001.9.5, , , , 00000000001.9.5, 00000000001.13.5, , ] 2020-11-19T03:34:17.442 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.9.5, , 00000000001.9.5, , , , 00000000001.9.5, 00000000001.13.5, , ] 2020-11-19T03:34:17.443 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:17.433+0000 7ff8d33f00c0 0 bucket checkpoint complete 2020-11-19T03:34:17.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-11 zones={test-zone3, test-zone2} 2020-11-19T03:34:17.782 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:17.787 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:17.788 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:17.788 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:17.789 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:17.789 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:17.795 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:17.795 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:17.796 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:17.796 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:17.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:17.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:17.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:18.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:18.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:18.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:18.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:18.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:18.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:18.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:18.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:18.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:19.061 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-11 zones={test-zone3, test-zone2} 2020-11-19T03:34:19.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:19.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:19.062 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:19.117 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:19.117 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:19.350 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:19.345+0000 7f74a4c4e0c0 1 bucket sync caught up with source: 2020-11-19T03:34:21.234 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.14.5, 00000000001.20.5, , ] 2020-11-19T03:34:21.234 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.14.5, 00000000001.20.5, , ] 2020-11-19T03:34:21.235 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:19.345+0000 7f74a4c4e0c0 0 bucket checkpoint complete 2020-11-19T03:34:21.236 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-12 zones={test-zone1, test-zone3} 2020-11-19T03:34:21.711 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:21.721 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:21.721 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:21.722 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:21.722 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:21.723 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:21.727 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:21.727 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:21.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:21.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:21.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:22.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:22.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:22.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:22.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:22.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:22.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:22.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:22.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:22.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:22.920 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-12 zones={test-zone1, test-zone3} 2020-11-19T03:34:22.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:22.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:22.922 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:22.970 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:22.970 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:23.211 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:23.211+0000 7f8015ea50c0 1 bucket sync caught up with source: 2020-11-19T03:34:23.211 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.14.5, 00000000001.20.5, , ] 2020-11-19T03:34:23.211 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.14.5, 00000000001.20.5, , ] 2020-11-19T03:34:23.211 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:23.211+0000 7f8015ea50c0 0 bucket checkpoint complete 2020-11-19T03:34:23.239 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-12 zones={test-zone2, test-zone3} 2020-11-19T03:34:23.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:23.926 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:23.927 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:23.927 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:23.928 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:23.928 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:23.935 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-19T03:34:23.935 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-19T03:34:23.936 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-19T03:34:23.936 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:23.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:24.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:24.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-19T03:34:24.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:24.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:24.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-19T03:34:25.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:25.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:25.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-19T03:34:25.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:25.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:25.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:25.831 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-12 zones={test-zone2, test-zone3} 2020-11-19T03:34:25.831 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-11-19T03:34:25.832 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kmbxto-13 2020-11-19T03:34:26.298 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kmbxto-14 2020-11-19T03:34:26.359 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kmbxto-15 2020-11-19T03:34:26.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:26.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:26.577 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:34:26.619 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:26.620 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:26.837 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:34:26.858 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:34:26.859 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:26.859 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:34:26.859 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:34:26.860 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:26.860 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:26.860 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:34:26.860 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:34:26.861 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:26.861 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:26.861 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:34:26.861 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:49.439575Z" 2020-11-19T03:34:26.862 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:26.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:26.862 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756866.398519_106.1", 2020-11-19T03:34:26.863 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:34:26.398519Z" 2020-11-19T03:34:26.863 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:26.863 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:34:26.864 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:34:26.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:26.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:26.866 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:34:26.908 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:26.909 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:27.135 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:34:27.135 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:34:27.135 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:34:27.135 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:34:27.136 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:34:27.136 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:34:27.136 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.137 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:34:27.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.137 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:34:27.137 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.137 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.138 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:34:27.138 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.138 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:34:27.138 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.139 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:34:27.139 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.139 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.140 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:34:27.140 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.140 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.140 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:34:27.140 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.141 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:27.141 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.141 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:34:27.142 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.142 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.143 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:34:27.143 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.143 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.143 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:34:27.144 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.144 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:27.144 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.144 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:34:27.144 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.145 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.145 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:34:27.145 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.146 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.146 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756866.398519_106.1", 2020-11-19T03:34:27.146 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.146 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:27.146 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.147 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:34:26.398519Z", 2020-11-19T03:34:27.147 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.147 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.147 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.148 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:34:27.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.148 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:34:27.148 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:34:27.149 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:34:27.149 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.149 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:34:27.150 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:34:27.151 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:34:27.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:27.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:27.152 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:34:27.193 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:27.194 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:27.438 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:34:27.438 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:34:27.438 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:34:27.439 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:34:27.439 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:34:27.440 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:34:27.440 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:27.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:27.441 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:34:27.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:27.441 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:34:27.442 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:27.442 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:27.442 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:34:27.442 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:27.442 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:34:27.443 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:27.443 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:34:27.443 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:27.443 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:27.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:27.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:27.444 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:34:27.444 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:27.445 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:27.445 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:34:27.445 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:27.445 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:34:27.446 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:27.446 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:34:27.446 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:27.446 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:27.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:27.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:27.447 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:34:27.447 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:27.447 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:27.447 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:34:27.448 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:27.448 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:34:27.448 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:27.448 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:34:27.448 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:27.449 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:27.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:27.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:34:27.449 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:34:27.449 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:34:27.450 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:34:27.450 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756866.398519_106.1", 2020-11-19T03:34:27.450 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:34:27.451 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:34:27.451 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:34:27.451 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:34:26.398519Z", 2020-11-19T03:34:27.451 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:34:27.451 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:27.452 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:27.452 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:34:27.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:34:27.452 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:34:27.453 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:34:27.453 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:34:27.453 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:34:27.453 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:34:27.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:34:27.455 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:34:27.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:27.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:34:27.457 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:34:27.498 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:27.499 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:27.766 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:34:27.767 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:34:27.767 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:34:27.767 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:34:27.767 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:34:27.768 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:34:27.768 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.768 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:34:27.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.769 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:34:27.769 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.769 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.769 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:34:27.770 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.770 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:34:27.770 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.770 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:34:27.771 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.771 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.772 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:34:27.772 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.773 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.773 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:34:27.773 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.773 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:27.774 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.774 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:34:27.774 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.775 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.775 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:34:27.775 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.776 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.776 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:34:27.776 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.776 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:27.776 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.780 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:34:27.780 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.780 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:34:27.781 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:34:27.781 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:34:27.782 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:34:27.782 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756866.398519_106.1", 2020-11-19T03:34:27.782 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:34:27.782 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:34:27.783 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:34:27.783 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:34:26.398519Z", 2020-11-19T03:34:27.783 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:34:27.783 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.784 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.784 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:34:27.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:34:27.784 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:34:27.784 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:34:27.785 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:34:27.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:34:27.785 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:34:27.786 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:34:27.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:27.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:27.788 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:27.892 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:27.892 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:28.115 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:28.115+0000 7f78564840c0 1 bucket sync caught up with source: 2020-11-19T03:34:28.116 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:28.116 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:28.116 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:28.115+0000 7f78564840c0 0 bucket checkpoint complete 2020-11-19T03:34:28.147 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-13 zones={test-zone2, test-zone1} 2020-11-19T03:34:28.306 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:28.315 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:28.315 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:28.362 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:28.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:28.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:28.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:28.697 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-13 zones={test-zone2, test-zone1} 2020-11-19T03:34:28.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:28.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:28.699 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:28.745 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:28.745 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:28.987 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:28.985+0000 7f2acf8c70c0 1 bucket sync caught up with source: 2020-11-19T03:34:28.987 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:28.988 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:28.988 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:28.985+0000 7f2acf8c70c0 0 bucket checkpoint complete 2020-11-19T03:34:29.013 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-13 zones={test-zone3, test-zone1} 2020-11-19T03:34:29.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:29.200 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:29.201 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:29.211 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:29.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:31.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:31.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:31.681 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-13 zones={test-zone3, test-zone1} 2020-11-19T03:34:31.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:31.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:31.682 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:31.729 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:31.729 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:31.969 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:31.971+0000 7f9fdd8180c0 1 bucket sync caught up with source: 2020-11-19T03:34:31.970 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:31.970 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:31.970 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:31.971+0000 7f9fdd8180c0 0 bucket checkpoint complete 2020-11-19T03:34:32.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:32.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:32.100 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:32.148 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:32.148 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:32.410 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:32.405+0000 7f197c2870c0 1 bucket sync caught up with source: 2020-11-19T03:34:32.410 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-11-19T03:34:32.411 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-11-19T03:34:32.411 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:32.405+0000 7f197c2870c0 0 bucket checkpoint complete 2020-11-19T03:34:32.442 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-14 zones={test-zone1, test-zone2} 2020-11-19T03:34:32.857 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:32.866 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:32.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:32.919 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:32.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:33.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:33.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:33.261 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-14 zones={test-zone1, test-zone2} 2020-11-19T03:34:33.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:33.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:33.263 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:33.312 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:33.312 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:33.553 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:33.545+0000 7f68114e70c0 1 bucket sync caught up with source: 2020-11-19T03:34:33.553 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-11-19T03:34:33.554 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-11-19T03:34:33.554 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:33.545+0000 7f68114e70c0 0 bucket checkpoint complete 2020-11-19T03:34:33.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-14 zones={test-zone3, test-zone2} 2020-11-19T03:34:33.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:33.684 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:33.684 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:33.694 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:33.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:33.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:33.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:34.045 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-14 zones={test-zone3, test-zone2} 2020-11-19T03:34:34.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:34.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:34.046 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:34.093 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:34.093 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:34.322 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:34.317+0000 7f897c4bb0c0 1 bucket sync caught up with source: 2020-11-19T03:34:35.784 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2020-11-19T03:34:35.784 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2020-11-19T03:34:35.784 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:34:34.317+0000 7f897c4bb0c0 0 bucket checkpoint complete 2020-11-19T03:34:35.785 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-15 zones={test-zone1, test-zone3} 2020-11-19T03:34:36.876 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:36.885 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:36.886 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:36.934 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:36.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:37.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:37.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:37.257 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-15 zones={test-zone1, test-zone3} 2020-11-19T03:34:37.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:37.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:37.259 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:37.308 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:37.309 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:37.566 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:37.563+0000 7fb9a10940c0 1 bucket sync caught up with source: 2020-11-19T03:34:37.566 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2020-11-19T03:34:37.567 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2020-11-19T03:34:37.567 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:37.563+0000 7fb9a10940c0 0 bucket checkpoint complete 2020-11-19T03:34:37.592 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-15 zones={test-zone2, test-zone3} 2020-11-19T03:34:38.746 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:34:38.755 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:38.756 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:34:38.814 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-19T03:34:38.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:38.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:39.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-19T03:34:39.183 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-15 zones={test-zone2, test-zone3} 2020-11-19T03:34:40.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:40.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:34:40.499 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:34:40.552 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:34:40.552 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:34:40.770 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:40.771+0000 7fba894270c0 1 waiting for incremental sync to catch up: 2020-11-19T03:34:40.771 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:40.771 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:34:45.774 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:45.775+0000 7fba894270c0 1 waiting for incremental sync to catch up: 2020-11-19T03:34:46.186 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:46.186 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:34:50.778 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:50.779+0000 7fba894270c0 1 waiting for incremental sync to catch up: 2020-11-19T03:34:51.054 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:51.054 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:34:55.782 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:34:55.783+0000 7fba894270c0 1 waiting for incremental sync to catch up: 2020-11-19T03:34:55.782 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:34:55.783 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:00.784 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:00.787+0000 7fba894270c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:01.106 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-19T03:35:01.107 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:05.788 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:05.791+0000 7fba894270c0 1 bucket sync caught up with source: 2020-11-19T03:35:06.508 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:06.508 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:06.508 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:05.791+0000 7fba894270c0 0 bucket checkpoint complete 2020-11-19T03:35:06.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-13 zones={test-zone2, test-zone1} 2020-11-19T03:35:07.504 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:35:07.514 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:35:07.627 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-13 zones={test-zone2, test-zone1} 2020-11-19T03:35:07.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:07.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:07.629 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:07.681 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:07.682 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:07.927 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:07.925+0000 7fdbd05370c0 1 bucket sync caught up with source: 2020-11-19T03:35:07.928 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:07.928 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:07.929 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:07.925+0000 7fdbd05370c0 0 bucket checkpoint complete 2020-11-19T03:35:07.956 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-13 zones={test-zone3, test-zone1} 2020-11-19T03:35:08.567 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:35:08.620 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:35:08.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-13 zones={test-zone3, test-zone1} 2020-11-19T03:35:08.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:08.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:08.689 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:08.740 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:08.740 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:08.985 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:08.987+0000 7f1e5444d0c0 1 bucket sync caught up with source: 2020-11-19T03:35:08.985 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:08.986 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2020-11-19T03:35:08.986 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:08.987+0000 7f1e5444d0c0 0 bucket checkpoint complete 2020-11-19T03:35:09.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:09.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:09.264 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:09.322 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:09.323 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:09.536 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:09.533+0000 7ff30ef780c0 1 bucket sync caught up with source: 2020-11-19T03:35:09.834 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T03:35:09.835 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T03:35:09.835 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:09.533+0000 7ff30ef780c0 0 bucket checkpoint complete 2020-11-19T03:35:09.836 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-14 zones={test-zone1, test-zone2} 2020-11-19T03:35:10.014 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:35:10.071 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:35:10.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-14 zones={test-zone1, test-zone2} 2020-11-19T03:35:10.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:10.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:10.147 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:10.203 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:10.203 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:10.430 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:10.425+0000 7fa5462bc0c0 1 bucket sync caught up with source: 2020-11-19T03:35:10.431 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T03:35:10.432 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000002.16.5, , , , , , , , , ] 2020-11-19T03:35:10.432 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:10.425+0000 7fa5462bc0c0 0 bucket checkpoint complete 2020-11-19T03:35:10.466 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-14 zones={test-zone3, test-zone2} 2020-11-19T03:35:10.691 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:35:10.738 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:35:10.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-14 zones={test-zone3, test-zone2} 2020-11-19T03:35:10.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:10.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:10.830 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:10.878 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:10.878 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:11.114 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:11.109+0000 7fdaa384b0c0 1 bucket sync caught up with source: 2020-11-19T03:35:11.114 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2020-11-19T03:35:11.114 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2020-11-19T03:35:11.115 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:11.109+0000 7fdaa384b0c0 0 bucket checkpoint complete 2020-11-19T03:35:11.149 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-15 zones={test-zone1, test-zone3} 2020-11-19T03:35:11.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:35:11.418 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:35:11.480 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-15 zones={test-zone1, test-zone3} 2020-11-19T03:35:11.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:11.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:11.482 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:11.530 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:11.530 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:11.752 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:11.755+0000 7f69fda5e0c0 1 bucket sync caught up with source: 2020-11-19T03:35:11.753 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2020-11-19T03:35:11.753 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2020-11-19T03:35:11.753 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:11.755+0000 7f69fda5e0c0 0 bucket checkpoint complete 2020-11-19T03:35:11.780 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-15 zones={test-zone2, test-zone3} 2020-11-19T03:35:11.791 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:35:11.842 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:35:11.904 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-15 zones={test-zone2, test-zone3} 2020-11-19T03:35:11.905 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-11-19T03:35:11.908 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kmbxto-16 2020-11-19T03:35:11.950 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kmbxto-17 2020-11-19T03:35:12.022 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kmbxto-18 2020-11-19T03:35:12.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:12.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:12.163 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:35:12.207 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:12.208 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:12.421 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:35:12.445 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:35:12.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:12.446 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:35:12.447 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:35:12.447 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:12.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:12.448 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:35:12.449 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:35:12.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:12.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:12.450 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:35:12.451 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:49.439575Z" 2020-11-19T03:35:12.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:12.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:12.452 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756912.144227_124.1", 2020-11-19T03:35:12.453 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:35:12.144227Z" 2020-11-19T03:35:12.454 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:35:12.454 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:35:12.457 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:35:12.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:12.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:12.461 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:35:12.506 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:12.507 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:12.754 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:35:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:35:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:35:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:35:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:35:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:35:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:35:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:35:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:35:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:35:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:35:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:35:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:35:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:35:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:35:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:35:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:35:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:35:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:35:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:35:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756912.144227_124.1", 2020-11-19T03:35:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:35:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:35:12.144227Z", 2020-11-19T03:35:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:35:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:35:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:35:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:35:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:12.767 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:35:12.768 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:35:12.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:35:12.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:12.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:12.770 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:35:12.812 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:12.812 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:13.082 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:35:13.082 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:35:13.082 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:35:13.082 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:35:13.083 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:35:13.083 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:35:13.083 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:35:13.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:13.084 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:35:13.084 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:13.084 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:35:13.084 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:35:13.084 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:35:13.085 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:35:13.088 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:35:13.088 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:35:13.088 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:35:13.089 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:35:13.089 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:35:13.089 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:35:13.089 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:13.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:13.090 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:35:13.090 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:35:13.090 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:35:13.090 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:35:13.091 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:35:13.091 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:35:13.091 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:35:13.091 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:35:13.091 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:35:13.092 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:35:13.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:13.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:13.092 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:35:13.092 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:35:13.093 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:35:13.093 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:35:13.093 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:35:13.093 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:35:13.093 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:35:13.094 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:35:13.094 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:35:13.094 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:35:13.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:13.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:35:13.095 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:35:13.095 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:35:13.095 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:35:13.095 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756912.144227_124.1", 2020-11-19T03:35:13.096 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:35:13.096 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:35:13.096 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:35:13.096 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:35:12.144227Z", 2020-11-19T03:35:13.096 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:35:13.097 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:35:13.097 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:35:13.097 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:35:13.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:35:13.097 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:35:13.098 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:35:13.098 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:35:13.098 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:35:13.098 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:35:13.099 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:35:13.099 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:35:13.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:13.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:35:13.101 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:35:13.152 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:13.153 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:13.419 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:35:13.420 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:35:13.420 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:35:13.420 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:35:13.420 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:35:13.420 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:35:13.421 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:13.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:13.421 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:35:13.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:13.422 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:35:13.422 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:13.422 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:13.422 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:35:13.423 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:13.423 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:35:13.423 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:13.423 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:35:13.423 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:13.424 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:13.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:13.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:13.424 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:35:13.424 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:13.425 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:13.425 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:35:13.425 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:13.425 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:35:13.425 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:13.426 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:35:13.426 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:13.426 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:13.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:13.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:13.427 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:35:13.427 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:13.427 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:13.427 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:35:13.428 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:13.428 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:35:13.428 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:13.428 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:35:13.428 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:13.429 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:13.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:13.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:35:13.429 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:35:13.429 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:35:13.430 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:35:13.430 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756912.144227_124.1", 2020-11-19T03:35:13.430 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:35:13.430 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:35:13.430 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:35:13.431 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:35:12.144227Z", 2020-11-19T03:35:13.431 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:35:13.431 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:13.431 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:13.432 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:35:13.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:35:13.432 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:35:13.432 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:35:13.432 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:35:13.433 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:35:13.433 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:35:13.434 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:35:13.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:13.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:13.566 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:13.617 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:13.618 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:13.859 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:13.863+0000 7f0f7abf30c0 1 bucket sync caught up with source: 2020-11-19T03:35:13.859 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:13.860 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:13.860 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:13.863+0000 7f0f7abf30c0 0 bucket checkpoint complete 2020-11-19T03:35:13.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:13.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:13.891 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:13.943 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:13.944 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:14.183 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:14.177+0000 7f9a8d92c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:14.183 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:14.184 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:19.184 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:19.181+0000 7f9a8d92c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:19.185 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:19.185 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:24.189 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:24.185+0000 7f9a8d92c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:25.258 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:25.259 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:29.193 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:29.188+0000 7f9a8d92c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:29.433 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:29.434 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:34.201 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:34.192+0000 7f9a8d92c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:34.861 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:34.861 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:39.204 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:39.200+0000 7f9a8d92c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:39.205 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:35:39.205 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:44.208 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:44.204+0000 7f9a8d92c0c0 1 bucket sync caught up with source: 2020-11-19T03:35:44.209 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:44.209 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:44.209 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:44.204+0000 7f9a8d92c0c0 0 bucket checkpoint complete 2020-11-19T03:35:44.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:44.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:44.246 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:44.299 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:44.299 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:44.538 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:44.536+0000 7f39e050f0c0 1 bucket sync caught up with source: 2020-11-19T03:35:46.683 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:46.684 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-11-19T03:35:46.684 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:35:44.536+0000 7f39e050f0c0 0 bucket checkpoint complete 2020-11-19T03:35:47.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:47.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:47.641 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:47.689 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:47.689 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:47.923 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:47.916+0000 7fc986c1e0c0 1 bucket sync caught up with source: 2020-11-19T03:35:47.923 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T03:35:47.924 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T03:35:47.924 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:47.916+0000 7fc986c1e0c0 0 bucket checkpoint complete 2020-11-19T03:35:47.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:47.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:35:47.957 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:35:48.012 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:35:48.013 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:35:48.247 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:48.240+0000 7f95b7d8d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:48.247 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T03:35:48.248 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T03:35:53.252 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:53.247+0000 7f95b7d8d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:53.252 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T03:35:53.252 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T03:35:58.256 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:35:58.251+0000 7f95b7d8d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:35:58.257 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T03:35:58.257 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T03:36:03.261 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:03.255+0000 7f95b7d8d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:03.261 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-19T03:36:03.262 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T03:36:08.266 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:08.259+0000 7f95b7d8d0c0 1 bucket sync caught up with source: 2020-11-19T03:36:08.267 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T03:36:08.267 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-19T03:36:08.267 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:08.259+0000 7f95b7d8d0c0 0 bucket checkpoint complete 2020-11-19T03:36:09.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:09.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:09.170 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:09.224 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:09.225 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:09.463 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:09.455+0000 7f089b6380c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:09.463 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , , , , , , ] 2020-11-19T03:36:09.463 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000002.16.10, , , , , ] 2020-11-19T03:36:14.467 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:14.459+0000 7f089b6380c0 1 bucket sync caught up with source: 2020-11-19T03:36:14.467 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-19T03:36:14.468 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000002.16.10, , , , , ] 2020-11-19T03:36:14.468 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:14.459+0000 7f089b6380c0 0 bucket checkpoint complete 2020-11-19T03:36:14.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:14.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:14.500 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:14.548 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:14.548 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:14.788 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:14.784+0000 7fd29f7050c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:14.789 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-19T03:36:14.789 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.22.9, , , , , ] 2020-11-19T03:36:19.791 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:19.787+0000 7fd29f7050c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:19.791 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-19T03:36:19.792 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.22.9, , , , , ] 2020-11-19T03:36:24.795 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:24.791+0000 7fd29f7050c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:24.795 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-19T03:36:24.798 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.22.9, , , , , ] 2020-11-19T03:36:29.798 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:29.795+0000 7fd29f7050c0 1 bucket sync caught up with source: 2020-11-19T03:36:29.799 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.22.9, , , , , ] 2020-11-19T03:36:29.799 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.22.9, , , , , ] 2020-11-19T03:36:29.799 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:29.795+0000 7fd29f7050c0 0 bucket checkpoint complete 2020-11-19T03:36:35.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:35.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:35.450 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:35.504 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:35.504 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:35.724 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:35.718+0000 7fc4ff1ed0c0 1 bucket sync caught up with source: 2020-11-19T03:36:35.724 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.23.5, , 00000000003.19.9, , 00000000009.22.9, , , , , ] 2020-11-19T03:36:35.725 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.23.5, , 00000000003.19.9, , 00000000009.22.9, , , , , ] 2020-11-19T03:36:35.725 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:35.718+0000 7fc4ff1ed0c0 0 bucket checkpoint complete 2020-11-19T03:36:35.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:35.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:35.754 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:35.802 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:35.803 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:36.058 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:36.051+0000 7f883b63f0c0 1 bucket sync caught up with source: 2020-11-19T03:36:36.058 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000010.23.5, , 00000000003.19.9, , 00000000009.22.9, , , , , ] 2020-11-19T03:36:36.059 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000010.23.5, , 00000000003.19.9, , 00000000009.22.9, , , , , ] 2020-11-19T03:36:36.059 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:36.051+0000 7f883b63f0c0 0 bucket checkpoint complete 2020-11-19T03:36:36.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:36.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:36.085 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:36.131 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:36.132 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:36.374 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:36.370+0000 7fa50d4cd0c0 1 bucket sync caught up with source: 2020-11-19T03:36:36.375 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.23.5, , 00000000003.19.9, , 00000000009.22.9, , , , , ] 2020-11-19T03:36:36.375 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.23.5, , 00000000003.19.9, , 00000000009.22.9, , , , , ] 2020-11-19T03:36:36.375 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:36.370+0000 7fa50d4cd0c0 0 bucket checkpoint complete 2020-11-19T03:36:36.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:36.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:36.500 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:36.550 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:36.550 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:36.786 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:36.779+0000 7f4e5587d0c0 1 bucket sync caught up with source: 2020-11-19T03:36:36.786 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.10.9, , 00000000010.37.5, , 00000000009.26.5, , , , , ] 2020-11-19T03:36:36.786 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.10.9, , 00000000010.37.5, , 00000000009.26.5, , , , , ] 2020-11-19T03:36:36.787 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:36.779+0000 7f4e5587d0c0 0 bucket checkpoint complete 2020-11-19T03:36:36.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:36.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:36.817 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:36.905 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:36.906 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:37.152 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:37.147+0000 7f8a127cd0c0 1 bucket sync caught up with source: 2020-11-19T03:36:37.153 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.10.9, , 00000000010.37.5, , 00000000009.26.5, , , , , ] 2020-11-19T03:36:37.153 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.10.9, , 00000000010.37.5, , 00000000009.26.5, , , , , ] 2020-11-19T03:36:37.153 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:37.147+0000 7f8a127cd0c0 0 bucket checkpoint complete 2020-11-19T03:36:37.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:37.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:37.187 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:37.235 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:37.235 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:37.465 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:37.459+0000 7f925c8540c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:37.466 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000002.9.10, , 00000000002.20.10, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:37.466 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.11.9, , 00000000003.22.9, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:42.471 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:42.463+0000 7f925c8540c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:42.471 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000002.9.10, , 00000000002.20.10, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:42.471 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.11.9, , 00000000003.22.9, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:47.475 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:47.467+0000 7f925c8540c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:47.476 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000002.9.10, , 00000000002.20.10, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:47.476 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.11.9, , 00000000003.22.9, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:52.479 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:52.471+0000 7f925c8540c0 1 bucket sync caught up with source: 2020-11-19T03:36:52.480 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.11.9, , 00000000003.22.9, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:52.480 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.11.9, , 00000000003.22.9, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:52.480 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:36:52.471+0000 7f925c8540c0 0 bucket checkpoint complete 2020-11-19T03:36:52.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:52.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:52.512 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:52.561 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:52.561 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:52.785 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:52.779+0000 7f793601b0c0 1 bucket sync caught up with source: 2020-11-19T03:36:52.785 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000003.11.9, , 00000000003.22.9, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:52.786 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000003.11.9, , 00000000003.22.9, , 00000000014.37.5, , , , , ] 2020-11-19T03:36:52.786 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:52.779+0000 7f793601b0c0 0 bucket checkpoint complete 2020-11-19T03:36:52.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-16 zones={test-zone2, test-zone1} 2020-11-19T03:36:53.112 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:36:53.120 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:36:53.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:36:53.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:36:53.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:36:53.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:36:53.130 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:36:53.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:36:53.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:36:53.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:36:53.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:53.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:53.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:53.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:53.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:53.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:53.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:53.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:53.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:53.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:53.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:54.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:54.239 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-16 zones={test-zone2, test-zone1} 2020-11-19T03:36:54.240 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-16 zones={test-zone3, test-zone1} 2020-11-19T03:36:54.977 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:36:54.987 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:36:54.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:36:54.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:36:54.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:36:54.989 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:36:54.995 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:36:54.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:36:54.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:36:54.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:36:55.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:55.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:55.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:55.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:55.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:55.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:55.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:55.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:55.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:55.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:55.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:55.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:56.041 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-16 zones={test-zone3, test-zone1} 2020-11-19T03:36:56.042 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-16 zones={test-zone3, test-zone2} 2020-11-19T03:36:56.855 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:36:56.863 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:36:56.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:36:56.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:36:56.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:36:56.865 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:36:56.871 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:36:56.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:36:56.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:36:56.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:36:56.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:56.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:56.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:36:57.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:57.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:57.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:36:57.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:57.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:57.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:36:57.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:57.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:57.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:36:57.957 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-16 zones={test-zone3, test-zone2} 2020-11-19T03:36:57.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:57.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:36:57.959 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:36:58.003 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:36:58.003 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:36:58.245 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:36:58.239+0000 7fcfb0a4d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:36:58.356 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000005.23.9, , 00000000004.12.9, , , , , ] 2020-11-19T03:36:58.356 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:03.249 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:03.242+0000 7fcfb0a4d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:03.250 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000005.23.9, , 00000000004.12.9, , , , , ] 2020-11-19T03:37:03.250 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:08.246+0000 7fcfb0a4d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:08.254 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000005.23.9, , 00000000004.12.9, , , , , ] 2020-11-19T03:37:08.254 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:13.258 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:13.249+0000 7fcfb0a4d0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:13.258 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000005.23.9, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:13.259 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:18.263 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:18.253+0000 7fcfb0a4d0c0 1 bucket sync caught up with source: 2020-11-19T03:37:18.264 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:18.264 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:18.264 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:18.253+0000 7fcfb0a4d0c0 0 bucket checkpoint complete 2020-11-19T03:37:18.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:18.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:18.293 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:18.347 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:18.347 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:18.575 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:18.567+0000 7f63430800c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:18.575 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:18.576 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:23.579 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:23.570+0000 7f63430800c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:23.579 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:23.579 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:28.585 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:28.574+0000 7f63430800c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:29.902 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:29.903 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:33.587 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:33.578+0000 7f63430800c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:33.587 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:33.587 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:38.590 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:38.582+0000 7f63430800c0 1 bucket sync caught up with source: 2020-11-19T03:37:38.591 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:38.591 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:38.591 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:38.582+0000 7f63430800c0 0 bucket checkpoint complete 2020-11-19T03:37:38.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:38.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:38.625 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:38.674 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:38.674 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:38.920 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:38.908+0000 7f9a0ae840c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:38.921 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:38.921 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:43.925 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:43.912+0000 7f9a0ae840c0 1 waiting for incremental sync to catch up: 2020-11-19T03:37:44.309 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000006.16.10, , , , , ] 2020-11-19T03:37:44.309 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:48.929 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:48.916+0000 7f9a0ae840c0 1 bucket sync caught up with source: 2020-11-19T03:37:48.929 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:48.930 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.31.5, , 00000000007.29.5, , 00000000007.18.9, , , , , ] 2020-11-19T03:37:48.930 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:48.916+0000 7f9a0ae840c0 0 bucket checkpoint complete 2020-11-19T03:37:49.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:49.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:49.058 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:49.116 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:49.116 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:49.365 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:49.358+0000 7f47054dc0c0 1 bucket sync caught up with source: 2020-11-19T03:37:49.365 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.19.9, , 00000000010.44.5, , 00000000009.24.9, , , , , ] 2020-11-19T03:37:49.366 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.19.9, , 00000000010.44.5, , 00000000009.24.9, , , , , ] 2020-11-19T03:37:49.366 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:49.358+0000 7f47054dc0c0 0 bucket checkpoint complete 2020-11-19T03:37:49.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:49.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:49.400 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:49.447 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:49.448 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:49.691 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:49.682+0000 7fabc433b0c0 1 bucket sync caught up with source: 2020-11-19T03:37:49.692 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.19.9, , 00000000010.44.5, , 00000000009.24.9, , , , , ] 2020-11-19T03:37:49.692 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.19.9, , 00000000010.44.5, , 00000000009.24.9, , , , , ] 2020-11-19T03:37:49.692 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:49.682+0000 7fabc433b0c0 0 bucket checkpoint complete 2020-11-19T03:37:49.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:49.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:49.727 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:49.831 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:49.831 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:50.084 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:50.078+0000 7f2826d1b0c0 1 bucket sync caught up with source: 2020-11-19T03:37:50.084 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.23.9, , 00000000007.34.5, , 00000000014.36.5, , , , , ] 2020-11-19T03:37:50.085 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.23.9, , 00000000007.34.5, , 00000000014.36.5, , , , , ] 2020-11-19T03:37:50.085 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:50.078+0000 7f2826d1b0c0 0 bucket checkpoint complete 2020-11-19T03:37:50.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:50.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:50.116 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:50.164 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:50.165 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:50.399 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:50.392+0000 7f01174100c0 1 bucket sync caught up with source: 2020-11-19T03:37:50.399 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000003.23.9, , 00000000007.34.5, , 00000000014.36.5, , , , , ] 2020-11-19T03:37:50.399 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000003.23.9, , 00000000007.34.5, , 00000000014.36.5, , , , , ] 2020-11-19T03:37:50.400 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:50.392+0000 7f01174100c0 0 bucket checkpoint complete 2020-11-19T03:37:50.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-17 zones={test-zone2, test-zone1} 2020-11-19T03:37:51.831 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:37:51.844 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:37:51.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:37:51.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:37:51.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:37:51.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:37:51.857 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:37:51.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:37:51.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:37:51.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:37:51.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:51.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:51.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:52.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:52.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:52.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:52.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:52.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:52.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:52.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:52.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:52.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:52.955 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-17 zones={test-zone2, test-zone1} 2020-11-19T03:37:52.956 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-17 zones={test-zone3, test-zone1} 2020-11-19T03:37:53.950 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:37:53.956 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:37:53.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:37:53.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:37:53.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:37:53.958 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:37:53.963 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:37:53.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:37:53.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:37:53.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:37:53.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:53.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:54.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:54.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:54.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:54.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:54.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:54.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:54.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:54.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:54.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:54.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:55.082 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-17 zones={test-zone3, test-zone1} 2020-11-19T03:37:55.083 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-17 zones={test-zone3, test-zone2} 2020-11-19T03:37:55.975 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:37:55.985 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:37:55.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:37:55.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:37:55.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:37:55.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:37:55.995 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:37:55.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:37:55.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:37:55.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:37:56.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:56.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:56.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:37:56.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:56.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:56.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:37:56.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:56.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:56.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:37:56.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:56.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:56.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:37:57.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-17 zones={test-zone3, test-zone2} 2020-11-19T03:37:57.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:57.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:57.040 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:57.092 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:57.092 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:57.321 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:57.311+0000 7f761bfb50c0 1 bucket sync caught up with source: 2020-11-19T03:37:57.559 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.21.9, , , , , ] 2020-11-19T03:37:57.560 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.21.9, , , , , ] 2020-11-19T03:37:57.560 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:57.311+0000 7f761bfb50c0 0 bucket checkpoint complete 2020-11-19T03:37:57.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:57.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:57.562 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:57.625 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:57.626 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:57.865 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:57.858+0000 7f61be3c10c0 1 bucket sync caught up with source: 2020-11-19T03:37:57.866 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.21.9, , , , , ] 2020-11-19T03:37:57.866 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.21.9, , , , , ] 2020-11-19T03:37:57.866 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:57.858+0000 7f61be3c10c0 0 bucket checkpoint complete 2020-11-19T03:37:57.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:57.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:57.893 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:57.942 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:57.942 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:58.191 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:58.183+0000 7f3a1b7880c0 1 bucket sync caught up with source: 2020-11-19T03:37:58.192 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.21.9, , , , , ] 2020-11-19T03:37:58.192 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.21.9, , , , , ] 2020-11-19T03:37:58.192 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:58.183+0000 7f3a1b7880c0 0 bucket checkpoint complete 2020-11-19T03:37:58.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:58.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:58.318 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:58.369 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:58.369 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:58.630 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:58.622+0000 7fe5b50b10c0 1 bucket sync caught up with source: 2020-11-19T03:37:58.630 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.20.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-19T03:37:58.630 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.20.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-19T03:37:58.630 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:58.622+0000 7fe5b50b10c0 0 bucket checkpoint complete 2020-11-19T03:37:58.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:58.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:58.664 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:58.755 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:58.755 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:58.984 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:58.978+0000 7f50d2fd90c0 1 bucket sync caught up with source: 2020-11-19T03:37:58.984 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.20.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-19T03:37:58.985 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.20.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-19T03:37:58.985 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:58.978+0000 7f50d2fd90c0 0 bucket checkpoint complete 2020-11-19T03:37:59.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:59.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:59.015 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:59.064 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:59.064 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:59.306 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:59.298+0000 7f4522f6e0c0 1 bucket sync caught up with source: 2020-11-19T03:37:59.307 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000003.17.9, , 00000000003.12.9, , 00000000016.44.5, , , , , ] 2020-11-19T03:37:59.307 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000003.17.9, , 00000000003.12.9, , 00000000016.44.5, , , , , ] 2020-11-19T03:37:59.307 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:37:59.298+0000 7f4522f6e0c0 0 bucket checkpoint complete 2020-11-19T03:37:59.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:59.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:37:59.343 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:37:59.391 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:37:59.391 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:37:59.618 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:59.611+0000 7fdcfa5580c0 1 bucket sync caught up with source: 2020-11-19T03:37:59.619 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000003.17.9, , 00000000003.12.9, , 00000000016.44.5, , , , , ] 2020-11-19T03:37:59.619 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000003.17.9, , 00000000003.12.9, , 00000000016.44.5, , , , , ] 2020-11-19T03:37:59.619 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:37:59.611+0000 7fdcfa5580c0 0 bucket checkpoint complete 2020-11-19T03:37:59.648 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-18 zones={test-zone2, test-zone1} 2020-11-19T03:38:00.352 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:00.359 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:00.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:00.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:00.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:00.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:00.367 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:00.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:00.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:00.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:00.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:00.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:00.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:00.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:00.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:00.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:00.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:00.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:01.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:01.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:01.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:01.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:01.418 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-18 zones={test-zone2, test-zone1} 2020-11-19T03:38:01.419 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-18 zones={test-zone3, test-zone1} 2020-11-19T03:38:02.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:02.203 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:02.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:02.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:02.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:02.204 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:02.211 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:02.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:02.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:02.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:02.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:02.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:02.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:02.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:02.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:02.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:02.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:02.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:02.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:02.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:03.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:03.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:05.758 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-18 zones={test-zone3, test-zone1} 2020-11-19T03:38:05.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-18 zones={test-zone3, test-zone2} 2020-11-19T03:38:05.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:05.914 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:05.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:05.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:05.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:05.916 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:05.923 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:05.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:05.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:05.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:05.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:05.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:06.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:06.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:06.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:06.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:06.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:06.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:06.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:06.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:06.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:06.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:06.925 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-18 zones={test-zone3, test-zone2} 2020-11-19T03:38:12.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:12.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:12.687 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:12.734 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:12.735 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:12.949 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:12.939+0000 7efc6033f0c0 1 bucket sync caught up with source: 2020-11-19T03:38:12.949 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000011.30.5, , 00000000004.22.5, , 00000000010.25.5, , , , , ] 2020-11-19T03:38:12.950 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000011.30.5, , 00000000004.22.5, , 00000000010.25.5, , , , , ] 2020-11-19T03:38:12.950 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:12.939+0000 7efc6033f0c0 0 bucket checkpoint complete 2020-11-19T03:38:12.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:12.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:12.980 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:13.032 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:13.033 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:13.284 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:13.278+0000 7f0b7b36d0c0 1 bucket sync caught up with source: 2020-11-19T03:38:13.284 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000011.30.5, , 00000000004.22.5, , 00000000010.25.5, , , , , ] 2020-11-19T03:38:13.285 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000011.30.5, , 00000000004.22.5, , 00000000010.25.5, , , , , ] 2020-11-19T03:38:13.285 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:13.278+0000 7f0b7b36d0c0 0 bucket checkpoint complete 2020-11-19T03:38:13.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:13.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:13.314 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:13.362 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:13.362 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:13.606 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:13.595+0000 7fe91cc3b0c0 1 bucket sync caught up with source: 2020-11-19T03:38:13.606 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000011.30.5, , 00000000004.22.5, , 00000000010.25.5, , , , , ] 2020-11-19T03:38:13.607 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000011.30.5, , 00000000004.22.5, , 00000000010.25.5, , , , , ] 2020-11-19T03:38:13.607 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:13.595+0000 7fe91cc3b0c0 0 bucket checkpoint complete 2020-11-19T03:38:13.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:13.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:13.737 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:13.794 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:13.795 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:14.038 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:14.030+0000 7f57cfdaf0c0 1 bucket sync caught up with source: 2020-11-19T03:38:14.038 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.13.5, , 00000000011.44.5, , 00000000010.30.5, , , , , ] 2020-11-19T03:38:14.038 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.13.5, , 00000000011.44.5, , 00000000010.30.5, , , , , ] 2020-11-19T03:38:14.038 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:14.030+0000 7f57cfdaf0c0 0 bucket checkpoint complete 2020-11-19T03:38:14.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:14.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:14.070 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:14.160 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:14.160 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:14.402 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:14.394+0000 7fdf201ce0c0 1 bucket sync caught up with source: 2020-11-19T03:38:14.403 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.13.5, , 00000000011.44.5, , 00000000010.30.5, , , , , ] 2020-11-19T03:38:14.403 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.13.5, , 00000000011.44.5, , 00000000010.30.5, , , , , ] 2020-11-19T03:38:14.403 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:14.394+0000 7fdf201ce0c0 0 bucket checkpoint complete 2020-11-19T03:38:14.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:14.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:14.436 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:14.484 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:14.484 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:14.725 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:14.718+0000 7f7bb813a0c0 1 bucket sync caught up with source: 2020-11-19T03:38:14.726 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.14.5, , 00000000004.25.5, , 00000000015.40.5, , , , , ] 2020-11-19T03:38:14.726 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.14.5, , 00000000004.25.5, , 00000000015.40.5, , , , , ] 2020-11-19T03:38:14.726 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:14.718+0000 7f7bb813a0c0 0 bucket checkpoint complete 2020-11-19T03:38:14.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:14.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:14.753 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:14.804 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:14.804 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:15.049 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:15.039+0000 7f3bb3ca20c0 1 bucket sync caught up with source: 2020-11-19T03:38:15.050 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000004.14.5, , 00000000004.25.5, , 00000000015.40.5, , , , , ] 2020-11-19T03:38:15.050 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000004.14.5, , 00000000004.25.5, , 00000000015.40.5, , , , , ] 2020-11-19T03:38:15.050 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:15.039+0000 7f3bb3ca20c0 0 bucket checkpoint complete 2020-11-19T03:38:15.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-16 zones={test-zone2, test-zone1} 2020-11-19T03:38:16.006 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:16.017 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:16.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:16.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:16.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:16.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:16.030 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:16.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:16.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:16.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:16.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:16.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:16.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:16.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:16.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:16.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:16.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:16.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:16.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:16.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:16.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:16.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:17.086 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-16 zones={test-zone2, test-zone1} 2020-11-19T03:38:17.088 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-16 zones={test-zone3, test-zone1} 2020-11-19T03:38:18.047 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:18.053 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:18.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:18.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:18.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:18.054 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:18.059 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:18.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:18.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:18.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:18.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:18.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:18.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:18.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:18.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:18.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:18.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:18.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:18.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:18.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:18.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:19.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:19.161 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-16 zones={test-zone3, test-zone1} 2020-11-19T03:38:19.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-16 zones={test-zone3, test-zone2} 2020-11-19T03:38:20.141 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:20.166 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:20.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:20.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:20.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:20.169 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:20.193 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:20.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:20.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:20.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:20.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:20.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:20.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:20.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:20.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:20.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:20.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:20.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:21.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:21.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:21.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:21.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:21.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-16 zones={test-zone3, test-zone2} 2020-11-19T03:38:21.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:21.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:21.467 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:21.511 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:21.512 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:21.733 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:21.723+0000 7fefad3ff0c0 1 bucket sync caught up with source: 2020-11-19T03:38:21.734 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000011.38.5, , 00000000008.33.5, , 00000000008.21.5, , , , , ] 2020-11-19T03:38:21.734 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000011.38.5, , 00000000008.33.5, , 00000000008.21.5, , , , , ] 2020-11-19T03:38:21.735 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:21.723+0000 7fefad3ff0c0 0 bucket checkpoint complete 2020-11-19T03:38:21.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:21.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:21.764 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:21.812 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:21.813 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:22.047 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:22.038+0000 7f87001ca0c0 1 bucket sync caught up with source: 2020-11-19T03:38:22.048 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000011.38.5, , 00000000008.33.5, , 00000000008.21.5, , , , , ] 2020-11-19T03:38:22.048 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000011.38.5, , 00000000008.33.5, , 00000000008.21.5, , , , , ] 2020-11-19T03:38:22.048 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:22.038+0000 7f87001ca0c0 0 bucket checkpoint complete 2020-11-19T03:38:22.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:22.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:22.075 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:22.123 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:22.124 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:22.375 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:22.367+0000 7fcfe7f9c0c0 1 bucket sync caught up with source: 2020-11-19T03:38:22.375 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000011.38.5, , 00000000008.33.5, , 00000000008.21.5, , , , , ] 2020-11-19T03:38:22.376 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000011.38.5, , 00000000008.33.5, , 00000000008.21.5, , , , , ] 2020-11-19T03:38:22.376 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:22.367+0000 7fcfe7f9c0c0 0 bucket checkpoint complete 2020-11-19T03:38:22.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:22.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:22.497 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:22.544 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:22.544 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:22.790 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:22.782+0000 7fd2995fa0c0 1 bucket sync caught up with source: 2020-11-19T03:38:22.791 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.22.5, , 00000000011.52.5, , 00000000010.27.5, , , , , ] 2020-11-19T03:38:22.791 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.22.5, , 00000000011.52.5, , 00000000010.27.5, , , , , ] 2020-11-19T03:38:22.791 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:22.782+0000 7fd2995fa0c0 0 bucket checkpoint complete 2020-11-19T03:38:22.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:22.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:22.823 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:22.912 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:22.912 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:23.173 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:23.166+0000 7f35e30990c0 1 bucket sync caught up with source: 2020-11-19T03:38:23.173 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.22.5, , 00000000011.52.5, , 00000000010.27.5, , , , , ] 2020-11-19T03:38:23.173 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.22.5, , 00000000011.52.5, , 00000000010.27.5, , , , , ] 2020-11-19T03:38:23.174 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:23.166+0000 7f35e30990c0 0 bucket checkpoint complete 2020-11-19T03:38:23.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:23.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:23.211 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:23.255 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:23.256 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:23.501 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:23.494+0000 7f00429040c0 1 bucket sync caught up with source: 2020-11-19T03:38:23.501 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.26.5, , 00000000008.37.5, , 00000000015.39.5, , , , , ] 2020-11-19T03:38:23.501 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.26.5, , 00000000008.37.5, , 00000000015.39.5, , , , , ] 2020-11-19T03:38:23.502 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:23.494+0000 7f00429040c0 0 bucket checkpoint complete 2020-11-19T03:38:23.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:23.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:23.529 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:23.579 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:23.579 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:23.808 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:23.799+0000 7f14268ff0c0 1 bucket sync caught up with source: 2020-11-19T03:38:23.809 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000004.26.5, , 00000000008.37.5, , 00000000015.39.5, , , , , ] 2020-11-19T03:38:23.809 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000004.26.5, , 00000000008.37.5, , 00000000015.39.5, , , , , ] 2020-11-19T03:38:23.811 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:23.799+0000 7f14268ff0c0 0 bucket checkpoint complete 2020-11-19T03:38:23.839 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-17 zones={test-zone2, test-zone1} 2020-11-19T03:38:24.796 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:24.806 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:24.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:24.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:24.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:24.808 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:24.817 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:24.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:24.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:24.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:24.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:24.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:24.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:24.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:25.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:25.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:25.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:25.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:25.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:25.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:25.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:25.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:25.917 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-17 zones={test-zone2, test-zone1} 2020-11-19T03:38:25.918 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-17 zones={test-zone3, test-zone1} 2020-11-19T03:38:26.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:26.812 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:26.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:26.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:26.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:26.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:26.823 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:26.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:26.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:26.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:26.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:26.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:26.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:27.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:27.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:27.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:27.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:27.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:27.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:27.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:27.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:27.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:27.898 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-17 zones={test-zone3, test-zone1} 2020-11-19T03:38:27.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-17 zones={test-zone3, test-zone2} 2020-11-19T03:38:28.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:28.347 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:28.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:28.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:28.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:28.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:28.356 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:28.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:28.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:28.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:28.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:28.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:28.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:28.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:28.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:28.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:28.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:28.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:29.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:29.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:29.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:29.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:29.441 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-17 zones={test-zone3, test-zone2} 2020-11-19T03:38:29.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:29.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:29.443 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:29.503 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:29.504 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:29.741 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:29.731+0000 7f1861b7a0c0 1 bucket sync caught up with source: 2020-11-19T03:38:29.742 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000011.32.5, , 00000000004.10.5, , 00000000008.24.5, , , , , ] 2020-11-19T03:38:29.742 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000011.32.5, , 00000000004.10.5, , 00000000008.24.5, , , , , ] 2020-11-19T03:38:29.742 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:29.731+0000 7f1861b7a0c0 0 bucket checkpoint complete 2020-11-19T03:38:29.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:29.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:29.781 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:29.833 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:29.833 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:30.084 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:30.078+0000 7f8031cdc0c0 1 bucket sync caught up with source: 2020-11-19T03:38:30.085 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000011.32.5, , 00000000004.10.5, , 00000000008.24.5, , , , , ] 2020-11-19T03:38:30.085 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000011.32.5, , 00000000004.10.5, , 00000000008.24.5, , , , , ] 2020-11-19T03:38:30.085 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:30.078+0000 7f8031cdc0c0 0 bucket checkpoint complete 2020-11-19T03:38:30.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:30.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:30.116 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:30.164 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:30.165 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:30.416 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:30.407+0000 7fe1a08fd0c0 1 bucket sync caught up with source: 2020-11-19T03:38:30.417 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000011.32.5, , 00000000004.10.5, , 00000000008.24.5, , , , , ] 2020-11-19T03:38:30.417 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000011.32.5, , 00000000004.10.5, , 00000000008.24.5, , , , , ] 2020-11-19T03:38:30.418 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:30.407+0000 7fe1a08fd0c0 0 bucket checkpoint complete 2020-11-19T03:38:30.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:30.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:30.535 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:30.586 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:30.586 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:30.828 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:30.818+0000 7fa63c2d10c0 1 bucket sync caught up with source: 2020-11-19T03:38:30.828 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.23.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-19T03:38:30.829 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.23.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-19T03:38:30.829 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:30.818+0000 7fa63c2d10c0 0 bucket checkpoint complete 2020-11-19T03:38:30.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:30.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:30.864 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:30.915 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:30.915 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:31.150 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:31.142+0000 7fcd1956a0c0 1 bucket sync caught up with source: 2020-11-19T03:38:31.150 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.23.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-19T03:38:31.151 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.23.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-19T03:38:31.151 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:31.142+0000 7fcd1956a0c0 0 bucket checkpoint complete 2020-11-19T03:38:31.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:31.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:31.183 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:31.238 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:31.238 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:31.460 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:31.450+0000 7f5680ba00c0 1 bucket sync caught up with source: 2020-11-19T03:38:31.460 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, 00000000004.20.5, , 00000000004.15.5, , 00000000017.47.5, , , , , ] 2020-11-19T03:38:31.461 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, 00000000004.20.5, , 00000000004.15.5, , 00000000017.47.5, , , , , ] 2020-11-19T03:38:31.461 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:31.450+0000 7f5680ba00c0 0 bucket checkpoint complete 2020-11-19T03:38:31.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:31.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:31.489 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:31.541 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:31.541 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:31.785 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:31.775+0000 7fd90830a0c0 1 bucket sync caught up with source: 2020-11-19T03:38:31.785 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, 00000000004.20.5, , 00000000004.15.5, , 00000000017.47.5, , , , , ] 2020-11-19T03:38:31.785 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, 00000000004.20.5, , 00000000004.15.5, , 00000000017.47.5, , , , , ] 2020-11-19T03:38:31.786 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:31.775+0000 7fd90830a0c0 0 bucket checkpoint complete 2020-11-19T03:38:31.815 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-18 zones={test-zone2, test-zone1} 2020-11-19T03:38:32.627 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:32.636 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:32.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:32.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:32.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:32.638 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:32.646 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:32.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:32.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:32.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:32.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:32.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:32.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:32.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:32.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:32.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:33.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:33.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:33.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:33.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:33.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:33.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:33.662 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-18 zones={test-zone2, test-zone1} 2020-11-19T03:38:33.663 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-18 zones={test-zone3, test-zone1} 2020-11-19T03:38:33.702 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:33.707 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:33.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:33.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:33.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:33.708 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:33.723 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:33.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:33.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:33.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:33.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:33.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:33.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:33.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:33.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:34.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:34.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:34.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:34.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:34.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:34.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:34.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:34.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-18 zones={test-zone3, test-zone1} 2020-11-19T03:38:34.718 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-18 zones={test-zone3, test-zone2} 2020-11-19T03:38:34.854 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:38:34.866 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:34.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:34.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:34.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:34.868 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:38:34.897 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:38:34.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-19T03:38:34.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-19T03:38:34.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-19T03:38:34.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:34.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:35.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:38:35.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:35.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:35.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-19T03:38:35.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:35.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:35.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-19T03:38:35.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:35.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:35.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-19T03:38:36.095 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-18 zones={test-zone3, test-zone2} 2020-11-19T03:38:36.097 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-11-19T03:38:36.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:36.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:36.847 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:38:36.884 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:36.885 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:37.092 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:38:37.127 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:38:37.127 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.127 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:38:37.127 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:38:37.128 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.128 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.128 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:38:37.128 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:38:37.128 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.129 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.129 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:38:37.129 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:49.439575Z" 2020-11-19T03:38:37.129 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.130 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.130 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605757116.840793_142.1", 2020-11-19T03:38:37.130 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:38:36.840793Z" 2020-11-19T03:38:37.130 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:38:37.130 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:38:37.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:38:37.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:37.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:37.133 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:38:37.178 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:37.179 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:37.441 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:38:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:38:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:38:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:38:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:38:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:38:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:38:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:38:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:38:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:38:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:38:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:38:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:38:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:38:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:38:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:38:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:38:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:38:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:38:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:38:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605757116.840793_142.1", 2020-11-19T03:38:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:38:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:38:36.840793Z", 2020-11-19T03:38:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:38:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:38:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:38:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:38:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:37.454 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:38:37.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:38:37.456 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:38:37.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:37.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:37.457 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:38:37.498 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:37.498 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:37.772 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:38:37.773 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:38:37.773 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:38:37.773 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:38:37.774 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:38:37.774 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:38:37.774 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:38:37.774 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.774 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:38:37.775 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.775 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:38:37.775 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:38:37.775 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:38:37.776 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:38:37.776 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:38:37.776 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:38:37.776 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:38:37.776 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:38:37.777 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:38:37.777 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:38:37.777 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.777 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.778 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:38:37.778 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:38:37.778 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:38:37.778 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:38:37.779 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:38:37.779 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:38:37.779 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:38:37.779 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:38:37.779 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:38:37.780 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:38:37.780 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.780 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.780 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:38:37.780 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:38:37.781 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:38:37.781 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:38:37.781 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:38:37.781 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:38:37.781 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:38:37.782 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:38:37.782 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:38:37.782 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:38:37.782 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.782 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:38:37.783 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:38:37.783 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:38:37.783 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:38:37.783 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605757116.840793_142.1", 2020-11-19T03:38:37.784 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:38:37.784 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:38:37.784 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:38:37.784 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:38:36.840793Z", 2020-11-19T03:38:37.784 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:38:37.785 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:38:37.785 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:38:37.785 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:38:37.785 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:38:37.785 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:38:37.786 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:38:37.786 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:38:37.786 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:38:37.786 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:38:37.788 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:38:37.789 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:38:37.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:37.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:38:37.791 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:38:37.832 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:37.833 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:38.099 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:38:38.100 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:38:38.100 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:38:38.100 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:38:38.100 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:38:38.100 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:38:38.101 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:38.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:38.101 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:38:38.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:38.102 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:38:38.102 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:38.102 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:38.102 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:38:38.102 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:38.103 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:38:38.103 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:38.103 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:38:38.103 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:38.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:38.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:38.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:38.104 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:38:38.104 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:38.104 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:38.105 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:38:38.105 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:38.105 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:38:38.105 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:38.105 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:38:38.106 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:38.106 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:38.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:38.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:38.106 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:38:38.107 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:38.107 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:38.107 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:38:38.107 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:38.108 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:38:38.108 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:38.108 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:38:38.108 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:38.108 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:38.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:38.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:38:38.109 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:38:38.109 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:38:38.109 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:38:38.110 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605757116.840793_142.1", 2020-11-19T03:38:38.110 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:38:38.110 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:38:38.110 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:38:38.110 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:38:36.840793Z", 2020-11-19T03:38:38.111 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:38:38.111 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:38.111 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:38.111 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:38:38.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:38:38.112 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:38:38.112 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:38:38.112 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:38:38.112 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:38:38.112 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:38:38.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-19T03:38:38.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:38.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:38.852 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:38.902 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:38.903 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:39.132 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:39.123+0000 7ff6305880c0 1 bucket sync caught up with source: 2020-11-19T03:38:39.133 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:38:39.133 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:38:39.133 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:39.123+0000 7ff6305880c0 0 bucket checkpoint complete 2020-11-19T03:38:39.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:39.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:39.161 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:39.210 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:39.211 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:39.431 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:39.422+0000 7f6c48d2c0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:38:39.431 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-11-19T03:38:39.432 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.12.9, , , , , ] 2020-11-19T03:38:44.436 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:44.430+0000 7f6c48d2c0c0 1 bucket sync caught up with source: 2020-11-19T03:38:44.436 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.12.9, , , , , ] 2020-11-19T03:38:44.437 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.12.9, , , , , ] 2020-11-19T03:38:44.437 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:44.430+0000 7f6c48d2c0c0 0 bucket checkpoint complete 2020-11-19T03:38:44.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:44.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:44.470 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:44.518 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:44.519 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:44.766 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:44.758+0000 7fdcb52600c0 1 bucket sync caught up with source: 2020-11-19T03:38:44.766 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T03:38:44.766 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-11-19T03:38:44.766 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:38:44.758+0000 7fdcb52600c0 0 bucket checkpoint complete 2020-11-19T03:38:44.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:44.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:44.889 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:44.938 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:44.938 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:45.184 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:45.178+0000 7fdd942860c0 1 bucket sync caught up with source: 2020-11-19T03:38:45.185 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:38:45.186 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:38:45.187 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:45.178+0000 7fdd942860c0 0 bucket checkpoint complete 2020-11-19T03:38:45.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:45.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:45.228 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:45.295 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:45.296 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:45.541 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:45.530+0000 7fbd7bdc50c0 1 bucket sync caught up with source: 2020-11-19T03:38:45.541 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:38:45.541 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:38:45.542 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:45.530+0000 7fbd7bdc50c0 0 bucket checkpoint complete 2020-11-19T03:38:45.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:45.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:38:45.571 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:38:45.661 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:38:45.661 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:38:45.905 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:45.898+0000 7f56c31ef0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:38:45.905 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000003.14.9, , , , , ] 2020-11-19T03:38:45.906 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:38:50.910 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:50.902+0000 7f56c31ef0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:38:50.910 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000003.14.9, , , , , ] 2020-11-19T03:38:50.911 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:38:55.914 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:38:55.906+0000 7f56c31ef0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:38:55.915 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000003.14.9, , , , , ] 2020-11-19T03:38:55.915 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:39:00.919 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:00.910+0000 7f56c31ef0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:00.920 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000003.14.9, , , , , ] 2020-11-19T03:39:00.920 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:39:05.924 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:05.914+0000 7f56c31ef0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:05.924 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000003.14.9, , , , , ] 2020-11-19T03:39:05.924 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:39:10.928 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:10.922+0000 7f56c31ef0c0 1 bucket sync caught up with source: 2020-11-19T03:39:10.929 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:39:10.929 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:39:10.929 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:10.922+0000 7f56c31ef0c0 0 bucket checkpoint complete 2020-11-19T03:39:10.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:10.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:10.964 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:39:11.011 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:39:11.012 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:39:11.241 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:11.235+0000 7fc21a10c0c0 1 bucket sync caught up with source: 2020-11-19T03:39:11.241 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:39:11.242 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-19T03:39:11.242 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:11.235+0000 7fc21a10c0c0 0 bucket checkpoint complete 2020-11-19T03:39:11.272 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-19 zones={test-zone2, test-zone1} 2020-11-19T03:39:12.718 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:39:12.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:39:12.732 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:39:12.787 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:39:12.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:12.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:12.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:13.021 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-19 zones={test-zone2, test-zone1} 2020-11-19T03:39:13.023 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-19 zones={test-zone3, test-zone1} 2020-11-19T03:39:13.063 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:39:13.110 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:39:13.111 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:39:13.116 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:39:13.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:13.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:13.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:13.360 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-19 zones={test-zone3, test-zone1} 2020-11-19T03:39:13.361 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-19 zones={test-zone3, test-zone2} 2020-11-19T03:39:13.372 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:39:13.418 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:39:13.419 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:39:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:39:13.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:13.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:13.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:39:13.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-19 zones={test-zone3, test-zone2} 2020-11-19T03:39:15.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:16.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:16.358 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:39:16.407 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:39:16.408 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:39:16.646 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:16.639+0000 7fdf229380c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:16.647 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000007.22.11, , ] 2020-11-19T03:39:16.647 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000030.84.11, , ] 2020-11-19T03:39:21.650 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:21.643+0000 7fdf229380c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:21.967 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000007.22.11, , ] 2020-11-19T03:39:21.967 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000030.84.11, , ] 2020-11-19T03:39:26.654 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:26.647+0000 7fdf229380c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:26.655 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000007.22.11, , ] 2020-11-19T03:39:26.655 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000030.84.11, , ] 2020-11-19T03:39:31.658 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:31.651+0000 7fdf229380c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:32.840 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000007.22.11, , ] 2020-11-19T03:39:32.841 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000030.84.11, , ] 2020-11-19T03:39:36.662 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:36.655+0000 7fdf229380c0 1 bucket sync caught up with source: 2020-11-19T03:39:37.542 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000040.114.5, , ] 2020-11-19T03:39:37.542 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000030.84.11, , ] 2020-11-19T03:39:37.542 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:36.655+0000 7fdf229380c0 0 bucket checkpoint complete 2020-11-19T03:39:37.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:37.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:37.545 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:39:37.597 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:39:37.598 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:39:37.851 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:37.842+0000 7f1c0aa7d0c0 1 bucket sync caught up with source: 2020-11-19T03:39:37.851 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000044.123.9, , ] 2020-11-19T03:39:37.851 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000044.123.9, , ] 2020-11-19T03:39:37.852 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:37.842+0000 7f1c0aa7d0c0 0 bucket checkpoint complete 2020-11-19T03:39:37.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:37.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:37.881 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:39:37.932 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:39:37.933 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:39:38.178 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:38.171+0000 7ff93450b0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:38.178 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000030.84.11, , ] 2020-11-19T03:39:38.179 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000044.123.9, , ] 2020-11-19T03:39:43.182 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:43.175+0000 7ff93450b0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:43.977 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000030.84.11, , ] 2020-11-19T03:39:43.977 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000044.123.9, , ] 2020-11-19T03:39:48.186 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:48.179+0000 7ff93450b0c0 1 bucket sync caught up with source: 2020-11-19T03:39:48.186 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000005.14.9, , , 00000000044.123.9, , ] 2020-11-19T03:39:48.186 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000005.14.9, , , 00000000044.123.9, , ] 2020-11-19T03:39:48.187 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:39:48.179+0000 7ff93450b0c0 0 bucket checkpoint complete 2020-11-19T03:39:48.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:50.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:50.259 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:39:50.308 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:39:50.309 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:39:50.559 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:50.549+0000 7f29d0dbf0c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000036.104.5, , ] 2020-11-19T03:39:50.560 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000045.128.11, , ] 2020-11-19T03:39:55.563 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:55.553+0000 7f29d0dbf0c0 1 bucket sync caught up with source: 2020-11-19T03:39:55.622 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000046.130.9, , ] 2020-11-19T03:39:55.622 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000045.128.11, , ] 2020-11-19T03:39:55.622 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:55.553+0000 7f29d0dbf0c0 0 bucket checkpoint complete 2020-11-19T03:39:55.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:55.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:39:55.625 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:39:55.673 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:39:55.673 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:39:55.926 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:39:55.917+0000 7f3d540620c0 1 waiting for incremental sync to catch up: 2020-11-19T03:39:55.927 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000046.130.9, , ] 2020-11-19T03:39:55.927 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000047.132.9, , ] 2020-11-19T03:40:00.931 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:40:00.921+0000 7f3d540620c0 1 waiting for incremental sync to catch up: 2020-11-19T03:40:01.360 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000046.130.9, , ] 2020-11-19T03:40:01.361 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000047.132.9, , ] 2020-11-19T03:40:05.938 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:40:05.925+0000 7f3d540620c0 1 waiting for incremental sync to catch up: 2020-11-19T03:40:06.009 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000046.130.9, , ] 2020-11-19T03:40:06.009 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000047.132.9, , ] 2020-11-19T03:40:10.936 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:40:10.929+0000 7f3d540620c0 1 waiting for incremental sync to catch up: 2020-11-19T03:40:10.936 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000046.130.9, , ] 2020-11-19T03:40:10.937 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000047.132.9, , ] 2020-11-19T03:40:15.941 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:40:15.937+0000 7f3d540620c0 1 bucket sync caught up with source: 2020-11-19T03:40:15.942 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000047.132.9, , ] 2020-11-19T03:40:15.943 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000047.132.9, , ] 2020-11-19T03:40:15.944 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:40:15.937+0000 7f3d540620c0 0 bucket checkpoint complete 2020-11-19T03:40:15.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:40:15.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:40:15.981 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-19T03:40:16.034 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:16.034 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:16.281 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:40:16.277+0000 7fa3152ba0c0 1 bucket sync caught up with source: 2020-11-19T03:40:16.281 INFO:teuthology.orchestra.run.smithi066.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000051.128.9, , ] 2020-11-19T03:40:16.281 INFO:teuthology.orchestra.run.smithi066.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000051.128.9, , ] 2020-11-19T03:40:16.281 INFO:teuthology.orchestra.run.smithi066.stderr:2020-11-19T03:40:16.277+0000 7fa3152ba0c0 0 bucket checkpoint complete 2020-11-19T03:40:16.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:40:16.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kmbxto-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-19T03:40:16.308 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kmbxto-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-19T03:40:16.359 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:16.359 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:16.614 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:40:16.607+0000 7f198190d0c0 1 bucket sync caught up with source: 2020-11-19T03:40:16.614 INFO:teuthology.orchestra.run.smithi154.stderr: local status: [, , , , , 00000000004.16.9, , , 00000000051.128.9, , ] 2020-11-19T03:40:16.615 INFO:teuthology.orchestra.run.smithi154.stderr: remote markers: [, , , , , 00000000004.16.9, , , 00000000051.128.9, , ] 2020-11-19T03:40:16.615 INFO:teuthology.orchestra.run.smithi154.stderr:2020-11-19T03:40:16.607+0000 7f198190d0c0 0 bucket checkpoint complete 2020-11-19T03:40:16.645 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-19 zones={test-zone2, test-zone1} 2020-11-19T03:40:16.661 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:40:16.676 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:40:16.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:40:16.713 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:40:16.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:16.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:16.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:16.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:16.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:17.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:18.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:19.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:20.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:21.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:22.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:23.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:24.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:25.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.234 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-19 zones={test-zone2, test-zone1} 2020-11-19T03:40:26.235 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-19 zones={test-zone3, test-zone1} 2020-11-19T03:40:26.245 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:40:26.254 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:40:26.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.268 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:40:26.276 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:40:26.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:26.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:26.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:26.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:26.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:26.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:27.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:28.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:29.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:30.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:31.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:32.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:33.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:34.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:35.830 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-19 zones={test-zone3, test-zone1} 2020-11-19T03:40:35.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kmbxto-19 zones={test-zone3, test-zone2} 2020-11-19T03:40:36.397 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-19T03:40:36.407 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:40:36.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.420 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-19T03:40:36.429 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-19T03:40:36.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-19T03:40:36.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:36.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:36.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-19T03:40:36.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:36.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:36.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:36.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:37.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:38.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:40.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:40.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:40.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:41.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:42.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:43.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:43.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:43.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:44.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:45.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:46.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:47.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:48.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:48.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:48.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:48.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:48.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:48.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:48.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:49.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:50.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:50.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:50.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-19T03:40:50.401 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kmbxto-19 zones={test-zone3, test-zone2} 2020-11-19T03:40:50.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:50.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:50.403 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:50.445 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:50.445 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:50.705 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:40:50.705 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:50.705 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:50.706 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:40:50.706 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:50.706 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:50.706 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:50.707 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:38:12.421680Z" 2020-11-19T03:40:50.707 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:50.707 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:50.707 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:50.707 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:40:09.171369Z" 2020-11-19T03:40:50.708 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:50.708 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:50.708 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:40:50.709 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:39:03.156946Z" 2020-11-19T03:40:50.709 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:50.709 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:40:50.710 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:40:50.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:50.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:50.712 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:50.758 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:50.759 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:51.021 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:40:51.021 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:40:51.021 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:40:51.022 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:40:51.022 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:40:51.022 INFO:teuthology.orchestra.run.smithi154.stdout: "instance_id": 14555551320719445728 2020-11-19T03:40:51.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:51.023 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:40:51.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:51.023 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:40:51.023 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:51.023 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.024 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:51.024 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:51.024 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:51.024 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:51.025 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000" 2020-11-19T03:40:51.025 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:51.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:51.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:51.026 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:40:51.026 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:51.026 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.026 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:38:12.421680Z" 2020-11-19T03:40:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:51.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:51.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:51.028 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:40:51.028 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:51.028 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.029 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:51.029 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:51.029 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:51.029 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:51.030 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:40:09.171369Z" 2020-11-19T03:40:51.030 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:51.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:51.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:51.030 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:40:51.031 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:51.031 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.031 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:40:51.032 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:51.032 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:51.032 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:51.032 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:39:03.156946Z" 2020-11-19T03:40:51.032 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:51.033 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:51.033 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:40:51.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:51.033 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:40:51.033 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 0, 2020-11-19T03:40:51.034 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 0 2020-11-19T03:40:51.034 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:51.034 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:40:51.035 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-19T03:40:51.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:51.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:51.037 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:51.076 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:51.076 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:51.339 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:40:51.339 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.340 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:51.340 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:40:51.340 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.340 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.341 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:51.341 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:38:12.421680Z" 2020-11-19T03:40:51.341 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.341 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.341 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:51.342 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:40:09.171369Z" 2020-11-19T03:40:51.342 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.342 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.342 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:40:51.342 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:39:03.156946Z" 2020-11-19T03:40:51.343 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:51.343 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:40:51.343 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:40:51.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:51.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:51.345 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:51.426 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:51.427 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:51.689 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:40:51.690 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:40:51.690 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:40:51.690 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:40:51.690 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:40:51.691 INFO:teuthology.orchestra.run.smithi066.stdout: "instance_id": 3122697539509682667 2020-11-19T03:40:51.691 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.691 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:40:51.691 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.691 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:40:51.692 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:51.692 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.692 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:51.692 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:51.692 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:51.693 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:51.693 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000" 2020-11-19T03:40:51.693 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:51.693 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.693 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.694 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:40:51.694 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:51.694 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.694 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:51.695 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:51.695 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:51.695 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:51.695 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:38:12.421680Z" 2020-11-19T03:40:51.696 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:51.696 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.696 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.697 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:40:51.697 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:51.697 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.697 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:51.697 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:51.698 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:51.698 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:51.698 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:40:09.171369Z" 2020-11-19T03:40:51.698 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:51.698 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.699 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:51.699 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:40:51.699 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:51.699 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:51.699 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:40:51.700 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:51.701 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:51.701 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:51.702 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:39:03.156946Z" 2020-11-19T03:40:51.702 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:51.702 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:51.702 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:40:51.702 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:51.703 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:40:51.703 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 0, 2020-11-19T03:40:51.703 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 0 2020-11-19T03:40:51.703 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:51.703 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:40:51.704 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-19T03:40:51.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:51.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:51.706 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:51.785 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:51.785 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:52.067 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:40:52.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:52.068 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:52.068 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:40:52.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:52.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:52.069 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:52.069 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:38:12.421680Z" 2020-11-19T03:40:52.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:52.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:52.070 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:52.070 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:40:09.171369Z" 2020-11-19T03:40:52.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:52.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:52.070 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:40:52.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:39:03.156946Z" 2020-11-19T03:40:52.071 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:52.071 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:40:52.072 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:40:52.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:52.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:52.074 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:52.117 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:52.118 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:52.383 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:40:52.385 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:40:52.385 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:40:52.386 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:40:52.387 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:40:52.389 INFO:teuthology.orchestra.run.smithi154.stdout: "instance_id": 4518964475850322464 2020-11-19T03:40:52.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.395 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:40:52.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.397 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:40:52.398 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:52.400 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:52.400 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:52.401 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:52.401 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:52.401 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:52.401 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000" 2020-11-19T03:40:52.401 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:52.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.402 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:40:52.402 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:52.403 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:52.403 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:52.403 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:52.403 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:52.403 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:52.404 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:38:12.421680Z" 2020-11-19T03:40:52.404 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:52.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.404 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:40:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:52.405 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:40:09.171369Z" 2020-11-19T03:40:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:40:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-19T03:40:52.407 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:39:03.156946Z" 2020-11-19T03:40:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:52.408 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:52.409 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:40:52.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.411 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:40:52.411 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 0, 2020-11-19T03:40:52.412 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 0 2020-11-19T03:40:52.412 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:52.413 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:40:52.415 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-19T03:40:52.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:52.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:52.417 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:52.504 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:52.505 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:52.760 INFO:teuthology.orchestra.run.smithi154.stdout:[ 2020-11-19T03:40:52.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.760 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:52.761 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "0.000000" 2020-11-19T03:40:52.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.761 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-19T03:40:52.761 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "2020-11-19T03:38:12.385011Z" 2020-11-19T03:40:52.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.762 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:52.762 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "2020-11-19T03:40:09.318973Z" 2020-11-19T03:40:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-19T03:40:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "2020-11-19T03:39:03.303838Z" 2020-11-19T03:40:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:52.764 INFO:teuthology.orchestra.run.smithi154.stdout:] 2020-11-19T03:40:52.765 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:40:52.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:52.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:52.767 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:52.812 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:52.812 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:53.084 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:40:53.084 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:40:53.084 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:40:53.085 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:40:53.085 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:40:53.085 INFO:teuthology.orchestra.run.smithi066.stdout: "instance_id": 12990004098457460101 2020-11-19T03:40:53.085 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.085 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:40:53.086 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.086 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:40:53.086 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.086 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.087 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:53.087 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.087 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.087 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.088 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000" 2020-11-19T03:40:53.088 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.088 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.089 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:40:53.089 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.089 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.089 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-19T03:40:53.090 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.090 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.090 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.090 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:38:12.385011Z" 2020-11-19T03:40:53.090 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.091 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:40:53.091 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.092 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.092 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:53.093 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.094 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.094 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.094 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:40:09.318973Z" 2020-11-19T03:40:53.095 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.095 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:40:53.096 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.096 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.096 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-19T03:40:53.096 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.097 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.097 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.097 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:39:03.303838Z" 2020-11-19T03:40:53.097 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.098 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.098 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:40:53.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.098 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:40:53.098 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 0, 2020-11-19T03:40:53.099 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 0 2020-11-19T03:40:53.099 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.099 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:40:53.100 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-19T03:40:53.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:53.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:53.102 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:53.144 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:53.145 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:53.407 INFO:teuthology.orchestra.run.smithi154.stdout:[ 2020-11-19T03:40:53.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:53.408 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:53.408 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "0.000000" 2020-11-19T03:40:53.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:53.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:53.408 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-19T03:40:53.409 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "2020-11-19T03:38:12.385011Z" 2020-11-19T03:40:53.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:53.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:53.409 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:53.409 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "2020-11-19T03:40:09.318973Z" 2020-11-19T03:40:53.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:53.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:53.410 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-19T03:40:53.410 INFO:teuthology.orchestra.run.smithi154.stdout: "last_update": "2020-11-19T03:39:03.303838Z" 2020-11-19T03:40:53.411 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:53.411 INFO:teuthology.orchestra.run.smithi154.stdout:] 2020-11-19T03:40:53.412 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:40:53.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:53.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:53.413 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:53.461 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:53.462 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:53.726 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:40:53.726 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:40:53.726 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:40:53.726 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:40:53.727 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:40:53.727 INFO:teuthology.orchestra.run.smithi066.stdout: "instance_id": 17342061985062302481 2020-11-19T03:40:53.727 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.727 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:40:53.727 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.728 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:40:53.728 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.728 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.728 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:53.729 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.729 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.729 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.729 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000" 2020-11-19T03:40:53.729 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.730 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.730 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.730 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:40:53.730 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.730 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.731 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-19T03:40:53.731 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.731 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.731 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.731 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:38:12.385011Z" 2020-11-19T03:40:53.732 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.732 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.732 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.732 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:40:53.732 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.733 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.733 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:53.733 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.733 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.733 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.734 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:40:09.318973Z" 2020-11-19T03:40:53.734 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.734 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.734 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:53.734 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:40:53.735 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:53.735 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:53.735 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-19T03:40:53.735 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:53.735 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:53.736 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:53.736 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:39:03.303838Z" 2020-11-19T03:40:53.736 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.736 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.736 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:40:53.737 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:53.737 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:40:53.737 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 0, 2020-11-19T03:40:53.737 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 0 2020-11-19T03:40:53.737 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:53.738 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:40:53.739 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-19T03:40:53.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:53.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:53.740 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:53.822 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:53.822 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:54.101 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:40:54.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.102 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:54.102 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:40:54.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.103 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.103 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:54.103 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:38:12.420850Z" 2020-11-19T03:40:54.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.103 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.104 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:54.104 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:40:09.249161Z" 2020-11-19T03:40:54.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.105 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.105 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-19T03:40:54.105 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:39:03.236141Z" 2020-11-19T03:40:54.105 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.105 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:40:54.106 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:40:54.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:54.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:54.108 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:54.188 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:54.189 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:54.448 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:40:54.449 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:40:54.449 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:40:54.449 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:40:54.449 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:40:54.450 INFO:teuthology.orchestra.run.smithi066.stdout: "instance_id": 17184965003939177913 2020-11-19T03:40:54.450 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.450 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:40:54.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.450 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:40:54.451 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:54.451 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:54.451 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:54.451 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:54.451 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:54.452 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:54.452 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000" 2020-11-19T03:40:54.452 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.453 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:40:54.453 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:54.453 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:54.453 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:54.454 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:54.454 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:54.454 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:54.454 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:38:12.420850Z" 2020-11-19T03:40:54.454 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.455 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:40:54.455 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:54.455 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:54.456 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:54.456 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:54.456 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:54.456 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:54.456 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:40:09.249161Z" 2020-11-19T03:40:54.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.457 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.457 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.457 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:40:54.457 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:54.458 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "incremental-sync", 2020-11-19T03:40:54.458 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-19T03:40:54.458 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:54.458 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:54.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:54.459 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:39:03.236141Z" 2020-11-19T03:40:54.459 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.459 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.459 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:40:54.459 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.460 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:40:54.460 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 0, 2020-11-19T03:40:54.460 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 0 2020-11-19T03:40:54.460 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.460 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:40:54.461 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-19T03:40:54.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:54.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:54.462 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:54.536 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:54.537 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:54.798 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:40:54.839 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.840 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:54.840 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:40:54.840 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.840 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.841 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:54.841 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:38:12.420850Z" 2020-11-19T03:40:54.841 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.841 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.842 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:54.842 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:40:09.249161Z" 2020-11-19T03:40:54.842 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:54.842 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:54.842 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-19T03:40:54.843 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:39:03.236141Z" 2020-11-19T03:40:54.843 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:54.843 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:40:54.844 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:40:54.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:54.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:54.846 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:54.893 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:54.894 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:55.154 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:40:55.154 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:40:55.155 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:40:55.155 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:40:55.155 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:40:55.155 INFO:teuthology.orchestra.run.smithi154.stdout: "instance_id": 13543837735053654907 2020-11-19T03:40:55.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:55.156 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:40:55.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:55.157 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:40:55.157 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:55.157 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:55.157 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-19T03:40:55.157 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:55.158 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:55.158 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:55.158 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000" 2020-11-19T03:40:55.158 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:55.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:55.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:55.159 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:40:55.159 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:55.159 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000002507", 2020-11-19T03:40:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:38:12.420850Z" 2020-11-19T03:40:55.161 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:55.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:55.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:55.161 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:40:55.161 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:55.162 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:55.162 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-19T03:40:55.162 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:55.162 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:55.162 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:55.163 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:40:09.249161Z" 2020-11-19T03:40:55.163 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:55.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:55.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:55.164 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:40:55.164 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:55.164 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "incremental-sync", 2020-11-19T03:40:55.164 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-19T03:40:55.164 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:55.165 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:55.165 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:55.165 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:39:03.236141Z" 2020-11-19T03:40:55.165 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:55.165 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:55.166 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:40:55.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:55.166 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:40:55.167 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 0, 2020-11-19T03:40:55.167 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 0 2020-11-19T03:40:55.167 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:55.167 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:40:55.168 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-19T03:40:55.169 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-11-19T03:40:55.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:55.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:55.391 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:55.430 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:55.430 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:55.657 INFO:teuthology.orchestra.run.smithi066.stderr:No --period given, using current period=e0ae9617-a343-47d4-9d97-66e550e1c440 2020-11-19T03:40:55.689 INFO:teuthology.orchestra.run.smithi066.stdout:[ 2020-11-19T03:40:55.690 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:55.690 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:40:55.690 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:31:28.618934Z" 2020-11-19T03:40:55.690 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:55.691 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:55.691 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:40:55.691 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "0.000000" 2020-11-19T03:40:55.691 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:55.691 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:55.692 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:40:55.692 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:33:49.439575Z" 2020-11-19T03:40:55.692 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:55.692 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:55.692 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605757255.384110_148.1", 2020-11-19T03:40:55.693 INFO:teuthology.orchestra.run.smithi066.stdout: "last_update": "2020-11-19T03:40:55.384110Z" 2020-11-19T03:40:55.693 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:55.693 INFO:teuthology.orchestra.run.smithi066.stdout:] 2020-11-19T03:40:55.694 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-19T03:40:55.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:55.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:55.696 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:55.742 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:55.742 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:55.993 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2020-11-19T03:40:55.994 INFO:teuthology.orchestra.run.smithi154.stdout: "sync_status": { 2020-11-19T03:40:55.994 INFO:teuthology.orchestra.run.smithi154.stdout: "info": { 2020-11-19T03:40:55.994 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "sync", 2020-11-19T03:40:55.995 INFO:teuthology.orchestra.run.smithi154.stdout: "num_shards": 4, 2020-11-19T03:40:55.995 INFO:teuthology.orchestra.run.smithi154.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:40:55.995 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:40:55.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:55.995 INFO:teuthology.orchestra.run.smithi154.stdout: "markers": [ 2020-11-19T03:40:55.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:55.996 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 0, 2020-11-19T03:40:55.996 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:55.996 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:40:55.996 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:40:55.997 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:55.997 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 1, 2020-11-19T03:40:55.997 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:55.997 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:40:55.998 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:40:55.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:55.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:55.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:55.998 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 1, 2020-11-19T03:40:55.999 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:55.999 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:40:55.999 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "", 2020-11-19T03:40:55.999 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:55.999 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:56.000 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:56.000 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "0.000000", 2020-11-19T03:40:56.000 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:40:56.000 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:56.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:56.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:56.001 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 2, 2020-11-19T03:40:56.001 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:56.001 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:40:56.001 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:40:56.002 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:56.002 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:56.002 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:56.002 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:40:56.002 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:40:56.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:56.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:56.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-11-19T03:40:56.003 INFO:teuthology.orchestra.run.smithi154.stdout: "key": 3, 2020-11-19T03:40:56.003 INFO:teuthology.orchestra.run.smithi154.stdout: "val": { 2020-11-19T03:40:56.004 INFO:teuthology.orchestra.run.smithi154.stdout: "state": 1, 2020-11-19T03:40:56.004 INFO:teuthology.orchestra.run.smithi154.stdout: "marker": "1_1605757255.384110_148.1", 2020-11-19T03:40:56.004 INFO:teuthology.orchestra.run.smithi154.stdout: "next_step_marker": "", 2020-11-19T03:40:56.004 INFO:teuthology.orchestra.run.smithi154.stdout: "total_entries": 0, 2020-11-19T03:40:56.004 INFO:teuthology.orchestra.run.smithi154.stdout: "pos": 0, 2020-11-19T03:40:56.005 INFO:teuthology.orchestra.run.smithi154.stdout: "timestamp": "2020-11-19T03:40:55.384110Z", 2020-11-19T03:40:56.005 INFO:teuthology.orchestra.run.smithi154.stdout: "realm_epoch": 2 2020-11-19T03:40:56.005 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:56.005 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:56.005 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2020-11-19T03:40:56.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2020-11-19T03:40:56.006 INFO:teuthology.orchestra.run.smithi154.stdout: "full_sync": { 2020-11-19T03:40:56.006 INFO:teuthology.orchestra.run.smithi154.stdout: "total": 1, 2020-11-19T03:40:56.006 INFO:teuthology.orchestra.run.smithi154.stdout: "complete": 1 2020-11-19T03:40:56.006 INFO:teuthology.orchestra.run.smithi154.stdout: } 2020-11-19T03:40:56.007 INFO:teuthology.orchestra.run.smithi154.stdout:} 2020-11-19T03:40:56.008 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-19T03:40:56.008 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-19T03:40:56.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:56.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:56.010 INFO:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:56.055 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:56.055 INFO:teuthology.orchestra.run.smithi066.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T03:40:56.301 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2020-11-19T03:40:56.301 INFO:teuthology.orchestra.run.smithi066.stdout: "sync_status": { 2020-11-19T03:40:56.302 INFO:teuthology.orchestra.run.smithi066.stdout: "info": { 2020-11-19T03:40:56.302 INFO:teuthology.orchestra.run.smithi066.stdout: "status": "sync", 2020-11-19T03:40:56.302 INFO:teuthology.orchestra.run.smithi066.stdout: "num_shards": 4, 2020-11-19T03:40:56.302 INFO:teuthology.orchestra.run.smithi066.stdout: "period": "e0ae9617-a343-47d4-9d97-66e550e1c440", 2020-11-19T03:40:56.302 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:40:56.303 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:56.303 INFO:teuthology.orchestra.run.smithi066.stdout: "markers": [ 2020-11-19T03:40:56.303 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:56.303 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 0, 2020-11-19T03:40:56.303 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:56.304 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:40:56.304 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756688.618934_42.1", 2020-11-19T03:40:56.304 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:56.304 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 1, 2020-11-19T03:40:56.304 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:56.305 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:31:28.618934Z", 2020-11-19T03:40:56.305 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:40:56.305 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:56.305 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:56.306 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:56.306 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 1, 2020-11-19T03:40:56.306 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:56.306 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:40:56.306 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "", 2020-11-19T03:40:56.307 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:56.307 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:56.307 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:56.307 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "0.000000", 2020-11-19T03:40:56.307 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:40:56.308 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:56.308 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:56.308 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:56.308 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 2, 2020-11-19T03:40:56.308 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:56.309 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:40:56.309 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605756829.439575_206.1", 2020-11-19T03:40:56.309 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:56.309 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:56.309 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:56.310 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:33:49.439575Z", 2020-11-19T03:40:56.310 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:40:56.310 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:56.310 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:56.311 INFO:teuthology.orchestra.run.smithi066.stdout: { 2020-11-19T03:40:56.311 INFO:teuthology.orchestra.run.smithi066.stdout: "key": 3, 2020-11-19T03:40:56.311 INFO:teuthology.orchestra.run.smithi066.stdout: "val": { 2020-11-19T03:40:56.311 INFO:teuthology.orchestra.run.smithi066.stdout: "state": 1, 2020-11-19T03:40:56.311 INFO:teuthology.orchestra.run.smithi066.stdout: "marker": "1_1605757255.384110_148.1", 2020-11-19T03:40:56.312 INFO:teuthology.orchestra.run.smithi066.stdout: "next_step_marker": "", 2020-11-19T03:40:56.312 INFO:teuthology.orchestra.run.smithi066.stdout: "total_entries": 0, 2020-11-19T03:40:56.312 INFO:teuthology.orchestra.run.smithi066.stdout: "pos": 0, 2020-11-19T03:40:56.312 INFO:teuthology.orchestra.run.smithi066.stdout: "timestamp": "2020-11-19T03:40:55.384110Z", 2020-11-19T03:40:56.312 INFO:teuthology.orchestra.run.smithi066.stdout: "realm_epoch": 2 2020-11-19T03:40:56.313 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:56.313 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:56.313 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2020-11-19T03:40:56.313 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2020-11-19T03:40:56.313 INFO:teuthology.orchestra.run.smithi066.stdout: "full_sync": { 2020-11-19T03:40:56.314 INFO:teuthology.orchestra.run.smithi066.stdout: "total": 1, 2020-11-19T03:40:56.314 INFO:teuthology.orchestra.run.smithi066.stdout: "complete": 1 2020-11-19T03:40:56.314 INFO:teuthology.orchestra.run.smithi066.stdout: } 2020-11-19T03:40:56.314 INFO:teuthology.orchestra.run.smithi066.stdout:} 2020-11-19T03:40:56.315 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-19T03:40:56.316 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-19T03:40:56.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:56.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-19T03:40:56.317 INFO:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-19T03:40:56.361 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setuser ceph since I am not root 2020-11-19T03:40:56.362 INFO:teuthology.orchestra.run.smithi154.stderr:ignoring --setgroup ceph since I am not root 2020-11-19T15:06:39.950 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-19T15:06:39.956 DEBUG:teuthology.task.console_log:Killing console logger for smithi066 2020-11-19T15:06:39.956 DEBUG:teuthology.task.console_log:Killing console logger for smithi154 2020-11-19T15:06:39.957 DEBUG:teuthology.task.console_log:Killing console logger for smithi066 2020-11-19T15:06:39.957 DEBUG:teuthology.task.console_log:Killing console logger for smithi154 2020-11-19T15:06:39.958 DEBUG:teuthology.exit:Finished running handlers