2020-03-11T22:19:17.894 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:19:17.894 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.waNb0u.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848250 2020-03-11T22:19:17.919 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:19:17.979 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848250 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848250' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27427 2020-03-11T22:19:17.979 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:19:17.979 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:19:17.980 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:19:17.981 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:19:17.994 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:19:17.995 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:19:17.995 INFO:teuthology.packaging:ref: None 2020-03-11T22:19:17.995 INFO:teuthology.packaging:tag: None 2020-03-11T22:19:17.995 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:19:17.995 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:19:17.995 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:19:18.217 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:19:18.217 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:19:18.228 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:19:18.228 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:19:18.254 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:19:18.254 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:19:18.309 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:19:28.327 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:19:38.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:19:48.360 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:19:58.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:08.397 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:18.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:28.425 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:38.444 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848250', 'machine_type': 'smithi'} 2020-03-11T22:20:38.463 DEBUG:teuthology.lock.ops:locked ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:20:38.501 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:20:38.689 INFO:teuthology.provision.fog.smithi130:Scheduling deploy of centos 8.1 2020-03-11T22:20:39.032 INFO:teuthology.orchestra.console:Power off smithi130 2020-03-11T22:20:39.033 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:20:39.058 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:43.082 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:47.194 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:47.404 INFO:teuthology.orchestra.console:Power off for smithi130 completed 2020-03-11T22:20:47.506 INFO:teuthology.orchestra.console:Power on smithi130 2020-03-11T22:20:47.506 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:20:47.528 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:51.549 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:55.663 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:20:55.874 INFO:teuthology.orchestra.console:Power on for smithi130 completed 2020-03-11T22:20:55.974 INFO:teuthology.provision.fog.smithi130:Waiting for deploy to finish 2020-03-11T22:23:27.224 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:36.315 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:45.370 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:23:54.490 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:03.546 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:12.603 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:21.663 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:27.667 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:27.999 INFO:teuthology.orchestra.run.smithi130:> hostname 2020-03-11T22:25:28.291 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130.front.sepia.ceph.com 2020-03-11T22:25:28.291 INFO:teuthology.orchestra.run.smithi130:> grep smithi130.front.sepia.ceph.com /etc/hosts 2020-03-11T22:25:28.305 INFO:teuthology.orchestra.run.smithi130.stdout:172.21.15.130 smithi130.front.sepia.ceph.com smithi130 2020-03-11T22:25:28.305 INFO:teuthology.orchestra.run.smithi130:> sudo hostname smithi130 2020-03-11T22:25:28.405 INFO:teuthology.orchestra.run.smithi130:> sudo sed -i -e 's/smithi130.front.sepia.ceph.com/smithi130/g' /etc/hosts 2020-03-11T22:25:28.433 INFO:teuthology.orchestra.run.smithi130:> sudo sed -i -e 's/172.21.15.130/172.21.15.130/g' /etc/hosts 2020-03-11T22:25:28.500 INFO:teuthology.orchestra.run.smithi130:> sudo sed -i -e 's/smithi130.front.sepia.ceph.com/smithi130/g' /etc/hostname 2020-03-11T22:25:28.567 INFO:teuthology.orchestra.run.smithi130:> sudo hostname smithi130 2020-03-11T22:25:28.633 INFO:teuthology.provision.fog.smithi130:Deploy complete! 2020-03-11T22:25:28.635 DEBUG:teuthology.parallel:result is None 2020-03-11T22:25:28.742 INFO:teuthology.lock.ops:Checking smithi130.front.sepia.ceph.com 2020-03-11T22:25:28.743 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:25:28.766 INFO:teuthology.lock.ops:Updating [smithi130.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:25:28.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:25:28.767 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:28.833 INFO:teuthology.orchestra.run.smithi130:> true 2020-03-11T22:25:28.995 INFO:teuthology.orchestra.run.smithi130:> uname -m 2020-03-11T22:25:29.049 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2020-03-11T22:25:29.050 INFO:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2020-03-11T22:25:29.103 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Linux" 2020-03-11T22:25:29.104 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8 (Core)" 2020-03-11T22:25:29.104 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2020-03-11T22:25:29.104 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:25:29.104 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8" 2020-03-11T22:25:29.104 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:25:29.104 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:25:29.105 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2020-03-11T22:25:29.105 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:25:29.105 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:25:29.105 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:25:29.105 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:29.105 INFO:teuthology.orchestra.run.smithi130.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:25:29.105 INFO:teuthology.orchestra.run.smithi130.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:25:29.106 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:25:29.106 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:25:29.106 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:29.107 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2020-03-11T22:25:29.120 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:25:29.147 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi130.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOUp0uFs+nmoHR6dkoDcJKqkvhG6I26nQT9OecouGeTm2GZJpVbzpPseBFR38eBv578Vy9XjfH3tjstdawNW0GbxDrcdiCtblNCmQC4CiDrSl1RSABB3TCJ03+IFQQJH2+vEVd0izS4Xqh+PBt6hbSQzXK0T3sOUZ2XftlS/2HKOZBE9tDXCMi7PusJmFVztP5AmvW7QBoPG2Ud+YIvfMOp9eNKkEth855DcARq2V0Uzha42BKo1BcMXEta5oO/qvZ5w7qb+fLQaxOji29/wivIlcw8ZgK4FDatlaGxsHU5pZVwHusZAseKmvfuGPNQOWHlLeNtqZAHC7fHv+caC+HgxRszBKBKHmsJhd8oh4wjvTrZDKpGhEioDsq7NfCEnZLhA+jpXeE7SJMSV1IIkeOfP0clz+1KWzZlwZX+JJNYaDYY84GofmrFOoMjPKLDMBZxaUE5Jz/parxqW9ev/EFk80fYn7+WD32+Jq4fZrsJJyl65LSo/xvdRHuygUVzoU= 2020-03-11T22:25:29.147 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:25:29.223 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:25:29.229 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:25:29.240 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:25:29.250 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:25:29.266 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:20:38.453268', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi130.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHQSScH/JqXPgaPzYDpIwnYyYg4fKLocJKindduAix1UnYEyrL+6eI7dPwv5Y5Fn4WBxiUhbOib8u/4GnFWSGBA=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848250'} 2020-03-11T22:25:29.266 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:25:29.277 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:25:29.277 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:25:29.402 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:25:29.403 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:25:29.413 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:25:29.413 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:25:29.415 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:29.485 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:25:29.496 INFO:teuthology.orchestra.run.smithi130:> uname -m 2020-03-11T22:25:29.644 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2020-03-11T22:25:29.644 INFO:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2020-03-11T22:25:29.698 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Linux" 2020-03-11T22:25:29.698 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8 (Core)" 2020-03-11T22:25:29.698 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2020-03-11T22:25:29.699 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:25:29.699 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8" 2020-03-11T22:25:29.699 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:25:29.699 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:25:29.700 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2020-03-11T22:25:29.700 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:25:29.700 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:25:29.700 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:25:29.700 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:29.701 INFO:teuthology.orchestra.run.smithi130.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:25:29.701 INFO:teuthology.orchestra.run.smithi130.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:25:29.701 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:25:29.701 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:25:29.701 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:29.702 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2020-03-11T22:25:29.719 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:25:29.744 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:25:29.755 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:25:29.755 INFO:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:25:29.769 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:25:29.779 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:25:29.780 INFO:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:25:29.834 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:25:29.861 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:25:29.881 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:25:29.881 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:25:29.881 INFO:teuthology.orchestra.run.smithi130:> uname -r 2020-03-11T22:25:29.895 INFO:teuthology.orchestra.run.smithi130.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:25:29.895 INFO:teuthology.task.kernel:Running kernel on smithi130: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:25:29.895 INFO:teuthology.orchestra.run.smithi130:> sudo yum install -y kernel 2020-03-11T22:25:32.540 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1 day, 6:59:39 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:32.928 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:25:32.929 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:25:33.093 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:25:33.094 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:25:33.094 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:25:33.218 INFO:teuthology.orchestra.run.smithi130:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:25:33.603 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1 day, 6:59:40 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:33.946 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:25:33.947 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:25:33.947 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2020-03-11T22:25:33.947 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:25:33.947 INFO:teuthology.orchestra.run.smithi130.stdout:Reinstalling: 2020-03-11T22:25:33.947 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:25:33.947 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:25:33.947 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:33.948 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:25:33.948 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:25:33.948 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:33.948 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 3.1 M 2020-03-11T22:25:33.948 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 0 2020-03-11T22:25:33.948 INFO:teuthology.orchestra.run.smithi130.stderr:Operation aborted. 2020-03-11T22:25:34.031 INFO:teuthology.orchestra.run.smithi130.stdout:Is this ok [y/N]: 2020-03-11T22:25:34.032 INFO:teuthology.orchestra.run.smithi130:> sudo yum reinstall -y kernel || true 2020-03-11T22:25:34.416 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1 day, 6:59:41 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:25:34.782 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:25:34.784 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:25:34.784 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2020-03-11T22:25:34.784 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:25:34.784 INFO:teuthology.orchestra.run.smithi130.stdout:Reinstalling: 2020-03-11T22:25:34.784 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:25:34.784 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:25:34.784 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:34.785 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:25:34.785 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:25:34.785 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:34.785 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 3.1 M 2020-03-11T22:25:34.785 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 0 2020-03-11T22:25:34.785 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:25:34.866 INFO:teuthology.orchestra.run.smithi130.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 22 MB/s | 1.5 MB 00:00 2020-03-11T22:25:34.873 INFO:teuthology.orchestra.run.smithi130.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2020-03-11T22:25:34.874 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:25:34.875 INFO:teuthology.orchestra.run.smithi130.stdout:Total 34 MB/s | 3.1 MB 00:00 2020-03-11T22:25:34.897 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:25:35.102 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:25:35.102 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:25:35.183 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:25:35.184 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:25:35.317 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:25:35.454 INFO:teuthology.orchestra.run.smithi130.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:25:35.588 INFO:teuthology.orchestra.run.smithi130.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:25:35.589 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:35.723 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:35.723 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:25:35.946 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:25:35.947 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:25:35.947 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:25:35.947 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:25:36.184 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:25:36.184 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:36.184 INFO:teuthology.orchestra.run.smithi130.stdout:Reinstalled: 2020-03-11T22:25:36.184 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:25:36.184 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:36.185 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:25:36.308 INFO:teuthology.orchestra.run.smithi130:> rpm -q kernel --last | head -n 1 2020-03-11T22:25:36.405 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:25:35 PM UTC 2020-03-11T22:25:36.406 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:25:36.406 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-11T22:25:36.406 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-11T22:25:36.406 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-11T22:25:36.406 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-11T22:25:36.407 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2020-03-11T22:25:36.407 INFO:teuthology.orchestra.run.smithi130:> rpm -q kernel --last | head -n 1 2020-03-11T22:25:36.448 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:25:35 PM UTC 2020-03-11T22:25:36.450 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:25:36.450 INFO:teuthology.orchestra.run.smithi130:> sudo rpm -qi grub2-tools 2020-03-11T22:25:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Name : grub2-tools 2020-03-11T22:25:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Epoch : 1 2020-03-11T22:25:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Version : 2.02 2020-03-11T22:25:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Release : 78.el8 2020-03-11T22:25:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: x86_64 2020-03-11T22:25:36.522 INFO:teuthology.orchestra.run.smithi130.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-11T22:25:36.522 INFO:teuthology.orchestra.run.smithi130.stdout:Group : System Environment/Base 2020-03-11T22:25:36.522 INFO:teuthology.orchestra.run.smithi130.stdout:Size : 9192691 2020-03-11T22:25:36.522 INFO:teuthology.orchestra.run.smithi130.stdout:License : GPLv3+ 2020-03-11T22:25:36.522 INFO:teuthology.orchestra.run.smithi130.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-11T22:25:36.522 INFO:teuthology.orchestra.run.smithi130.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-11T22:25:36.522 INFO:teuthology.orchestra.run.smithi130.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-11T22:25:36.523 INFO:teuthology.orchestra.run.smithi130.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-11T22:25:36.523 INFO:teuthology.orchestra.run.smithi130.stdout:Relocations : (not relocatable) 2020-03-11T22:25:36.523 INFO:teuthology.orchestra.run.smithi130.stdout:Packager : CentOS BuildSystem 2020-03-11T22:25:36.523 INFO:teuthology.orchestra.run.smithi130.stdout:Vendor : CentOS 2020-03-11T22:25:36.523 INFO:teuthology.orchestra.run.smithi130.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-11T22:25:36.523 INFO:teuthology.orchestra.run.smithi130.stdout:Summary : Support tools for GRUB. 2020-03-11T22:25:36.523 INFO:teuthology.orchestra.run.smithi130.stdout:Description : 2020-03-11T22:25:36.524 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:36.524 INFO:teuthology.orchestra.run.smithi130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-11T22:25:36.524 INFO:teuthology.orchestra.run.smithi130.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-11T22:25:36.524 INFO:teuthology.orchestra.run.smithi130.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-11T22:25:36.524 INFO:teuthology.orchestra.run.smithi130.stdout:hardware devices. 2020-03-11T22:25:36.524 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:25:36.524 INFO:teuthology.orchestra.run.smithi130.stdout:This subpackage provides tools for support of all platforms. 2020-03-11T22:25:36.525 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-11T22:25:36.525 INFO:teuthology.task.kernel:Updating grub on smithi130 to boot 4.18.0-147.el8.x86_64 2020-03-11T22:25:36.527 INFO:teuthology.orchestra.run.smithi130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-11T22:25:37.423 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2020-03-11T22:25:39.293 INFO:teuthology.orchestra.run.smithi130.stderr:done 2020-03-11T22:25:39.295 INFO:teuthology.orchestra.run.smithi130:> mktemp 2020-03-11T22:25:39.327 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.XpilQ2z569 2020-03-11T22:25:39.327 INFO:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XpilQ2z569 2020-03-11T22:25:39.409 INFO:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.XpilQ2z569 2020-03-11T22:25:39.557 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.XpilQ2z569 is 4KB 2020-03-11T22:25:39.605 INFO:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.XpilQ2z569 2020-03-11T22:25:39.620 INFO:teuthology.orchestra.run.smithi130:> sudo /bin/ls /boot/loader/entries || true 2020-03-11T22:25:39.689 INFO:teuthology.orchestra.run.smithi130.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-11T22:25:39.690 INFO:teuthology.orchestra.run.smithi130.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-11T22:25:39.690 INFO:teuthology.orchestra.run.smithi130.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-11T22:25:39.690 INFO:teuthology.orchestra.run.smithi130:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-11T22:25:39.861 INFO:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2020-03-11T22:26:09.890 INFO:teuthology.misc:Re-opening connections... 2020-03-11T22:26:09.890 INFO:teuthology.misc:trying to connect to ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:26:09.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:09.891 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:28.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2020-03-11T22:26:28.454 DEBUG:teuthology.misc:waited 18.5637278557 2020-03-11T22:26:29.455 INFO:teuthology.misc:trying to connect to ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:26:29.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:29.455 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:32.522 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2020-03-11T22:26:32.522 DEBUG:teuthology.misc:waited 22.6316950321 2020-03-11T22:26:33.523 INFO:teuthology.misc:trying to connect to ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:26:33.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:33.524 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:35.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2020-03-11T22:26:35.590 DEBUG:teuthology.misc:waited 25.6997270584 2020-03-11T22:26:36.591 INFO:teuthology.misc:trying to connect to ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:26:36.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:36.592 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:38.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2020-03-11T22:26:38.662 DEBUG:teuthology.misc:waited 28.7716119289 2020-03-11T22:26:39.663 INFO:teuthology.misc:trying to connect to ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:26:39.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:39.663 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:39.665 DEBUG:teuthology.orchestra.remote:timed out 2020-03-11T22:27:39.665 DEBUG:teuthology.misc:waited 89.7749049664 2020-03-11T22:27:40.665 INFO:teuthology.misc:trying to connect to ubuntu@smithi130.front.sepia.ceph.com 2020-03-11T22:27:40.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:40.666 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:41.011 INFO:teuthology.orchestra.run.smithi130:> true 2020-03-11T22:27:41.319 DEBUG:teuthology.misc:waited 91.4283180237 2020-03-11T22:27:42.320 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-11T22:27:42.320 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-11T22:27:42.320 INFO:teuthology.orchestra.run.smithi130:> uname -r 2020-03-11T22:27:42.337 INFO:teuthology.orchestra.run.smithi130.stdout:4.18.0-147.el8.x86_64 2020-03-11T22:27:42.337 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-11T22:27:42.337 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-11T22:27:43.338 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:27:43.339 INFO:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:27:43.426 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2020-03-11T22:27:43.440 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:27:43.447 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:27:43.447 INFO:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:27:43.463 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:27:43.473 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:27:43.487 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:27:43.488 INFO:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:27:43.532 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:27:43.542 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:27:43.543 INFO:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:27:43.605 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:27:43.620 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:27:43.622 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:27:43.633 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:27:43.634 INFO:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:27:43.692 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:27:43.709 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:27:43.709 INFO:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:27:43.748 INFO:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:27:43.822 INFO:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:27:43.879 INFO:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:27:43.953 INFO:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:27:44.003 INFO:teuthology.orchestra.run.smithi130:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:27:44.089 INFO:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2020-03-11T22:27:44.195 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:27:44.511 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:27:44.527 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:27:44.528 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:27:44.624 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:27:44.640 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:27:44.640 INFO:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2020-03-11T22:27:44.691 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: [ OK ] 2020-03-11T22:27:44.692 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:27:44.693 INFO:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2020-03-11T22:27:44.709 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2020-03-11T22:27:44.709 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:27:44.710 INFO:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:27:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:27:44.779 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:27:44.779 INFO:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:27:44.862 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:27:44.887 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-11T22:27:44.887 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:27:45.062 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:27:45.067 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:27:45.848 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:27:48.083 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:27:51.604 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:27:51.646 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:51 +0000 (0:00:00.166) 0:00:00.166 ******* 2020-03-11T22:27:54.294 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:54.296 INFO:teuthology.task.ansible.out:TASK [Gathering Facts] ********************************************************* 2020-03-11T22:27:54.296 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] 2020-03-11T22:27:54.297 INFO:teuthology.task.ansible.out: 2020-03-11T22:27:54.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:54 +0000 (0:00:02.734) 0:00:02.900 ******* 2020-03-11T22:27:54.740 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:27:54.741 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:27:54.779 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:54 +0000 (0:00:00.398) 0:00:03.299 ******* 2020-03-11T22:27:55.406 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-11T22:27:55.406 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:27:55.449 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:00.670) 0:00:03.969 ******* 2020-03-11T22:27:55.475 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:27:55.476 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:55.518 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:00.069) 0:00:04.038 ******* 2020-03-11T22:27:55.864 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-11T22:27:55.865 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:27:55.925 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:00.406) 0:00:04.445 ******* 2020-03-11T22:27:56.961 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:27:56.963 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965675.95-141978557326118/source", "state": "file", "uid": 0} 2020-03-11T22:27:57.000 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:56 +0000 (0:00:01.074) 0:00:05.520 ******* 2020-03-11T22:27:57.453 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-11T22:27:57.455 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-11T22:27:57.456 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:27:57.558 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2020-03-11T22:27:57.560 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:27:57.566 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:27:57.764 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:57 +0000 (0:00:00.764) 0:00:06.284 ******* 2020-03-11T22:27:58.430 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi130.front.sepia.ceph.com] 2020-03-11T22:27:58.528 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-11T22:27:58.529 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:58 +0000 (0:00:00.763) 0:00:07.048 ******* 2020-03-11T22:27:59.448 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-11T22:27:59.449 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.448212", "end": "2020-03-11 22:27:59.419733", "rc": 0, "start": "2020-03-11 22:27:58.971521", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:27:59.485 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.957) 0:00:08.005 ******* 2020-03-11T22:27:59.709 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:27:59.710 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009444", "end": "2020-03-11 22:27:59.679453", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:27:59.670009", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:27:59.752 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.267) 0:00:08.273 ******* 2020-03-11T22:27:59.778 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:27:59.780 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:27:59.815 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.062) 0:00:08.335 ******* 2020-03-11T22:27:59.841 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:27:59.842 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:59.893 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.078) 0:00:08.413 ******* 2020-03-11T22:27:59.920 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:27:59.922 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:59.966 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:59 +0000 (0:00:00.072) 0:00:08.486 ******* 2020-03-11T22:27:59.992 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:27:59.994 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.041 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.074) 0:00:08.561 ******* 2020-03-11T22:28:00.067 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:28:00.069 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.108 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.067) 0:00:08.629 ******* 2020-03-11T22:28:00.146 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:28:00.148 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.189 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:00 +0000 (0:00:00.079) 0:00:08.708 ******* 2020-03-11T22:28:00.248 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:28:00.248 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.260 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.271 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.283 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.299 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ f 2020-03-11T22:28:00.300 INFO:teuthology.task.ansible.out:f4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.309 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.320 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.321 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.332 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.333 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.344 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.345 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.356 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.374 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.385 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.386 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.398 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.409 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.409 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.421 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.422 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.433 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.434 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.445 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.456 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.456 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.467 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.468 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.479 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.480 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.491 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.502 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.503 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.514 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.515 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.526 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.537 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.538 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.539 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.550 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.561 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.562 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.573 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.574 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.584 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.585 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.596 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.596 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.607 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.608 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.619 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.629 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.630 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.641 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.642 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.653 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.666 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.676 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.688 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.689 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.690 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.702 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.703 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.714 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.726 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.737 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.748 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.749 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.760 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.771 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.782 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.783 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.794 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.805 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.805 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.807 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.817 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.828 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.828 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.839 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.840 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.851 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.852 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.863 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.864 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.874 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.875 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.886 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.898 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.898 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.910 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.921 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.923 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.924 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.935 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.948 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.959 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.960 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.976 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.977 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.989 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:00.989 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.001 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.002 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.014 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.025 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.026 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.037 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.038 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.115 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.926) 0:00:09.635 ******* 2020-03-11T22:28:01.141 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:28:01.177 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.062) 0:00:09.698 ******* 2020-03-11T22:28:01.203 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-11T22:28:01.206 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.246 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.068) 0:00:09.766 ******* 2020-03-11T22:28:01.283 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-11T22:28:01.285 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.330 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.084) 0:00:09.850 ******* 2020-03-11T22:28:01.358 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-11T22:28:01.365 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.408 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.077) 0:00:09.928 ******* 2020-03-11T22:28:01.436 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-11T22:28:01.437 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.476 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.067) 0:00:09.996 ******* 2020-03-11T22:28:01.504 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-11T22:28:01.506 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.555 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.078) 0:00:10.075 ******* 2020-03-11T22:28:01.591 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-11T22:28:01.593 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.603 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.604 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.605 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.616 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.618 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.628 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.630 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.639 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.652 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.653 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.654 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.665 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.666 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.678 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.688 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.711 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.722 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.723 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.725 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.735 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.736 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.747 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.748 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.759 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.760 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.771 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.772 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.784 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.796 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.808 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.820 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.822 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.841 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:01.882 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:01 +0000 (0:00:00.327) 0:00:10.402 ******* 2020-03-11T22:28:02.709 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-11T22:28:02.710 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:28:02.751 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:02 +0000 (0:00:00.869) 0:00:11.271 ******* 2020-03-11T22:28:02.787 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:28:02.789 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => { "msg": "Host smithi130.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:28:02.830 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:02 +0000 (0:00:00.079) 0:00:11.350 ******* 2020-03-11T22:28:02.888 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-11T22:28:02.889 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:28:02.936 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:02 +0000 (0:00:00.106) 0:00:11.456 ******* 2020-03-11T22:28:02.965 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-11T22:28:02.966 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.003 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:02 +0000 (0:00:00.066) 0:00:11.523 ******* 2020-03-11T22:28:03.268 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:28:03.270 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.065467", "end": "2020-03-11 22:28:03.243733", "rc": 0, "start": "2020-03-11 22:28:03.178266", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:28:03.308 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.304) 0:00:11.828 ******* 2020-03-11T22:28:03.347 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-11T22:28:03.348 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.399 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.090) 0:00:11.919 ******* 2020-03-11T22:28:03.426 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:28:03.427 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.463 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.064) 0:00:11.983 ******* 2020-03-11T22:28:03.488 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:28:03.490 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.536 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.072) 0:00:12.056 ******* 2020-03-11T22:28:03.562 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:28:03.564 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.611 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.075) 0:00:12.131 ******* 2020-03-11T22:28:03.638 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-11T22:28:03.639 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.681 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.069) 0:00:12.201 ******* 2020-03-11T22:28:03.718 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:28:03.719 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.766 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.084) 0:00:12.286 ******* 2020-03-11T22:28:03.806 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:28:03.808 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi130.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:28:03.845 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.079) 0:00:12.366 ******* 2020-03-11T22:28:03.883 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-11T22:28:03.884 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.922 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.076) 0:00:12.442 ******* 2020-03-11T22:28:03.948 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-11T22:28:03.950 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.987 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.064) 0:00:12.507 ******* 2020-03-11T22:28:04.023 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:28:04.024 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:28:04.071 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.084) 0:00:12.591 ******* 2020-03-11T22:28:04.097 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:28:04.098 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.137 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.065) 0:00:12.657 ******* 2020-03-11T22:28:04.163 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:28:04.164 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:28:04.204 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.065) 0:00:12.723 ******* 2020-03-11T22:28:04.229 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-11T22:28:04.230 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.277 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.074) 0:00:12.797 ******* 2020-03-11T22:28:04.314 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:28:04.316 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.356 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.078) 0:00:12.876 ******* 2020-03-11T22:28:04.395 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:28:04.397 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.440 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.084) 0:00:12.960 ******* 2020-03-11T22:28:04.479 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:28:04.480 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.528 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.087) 0:00:13.048 ******* 2020-03-11T22:28:04.556 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:28:04.558 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:28:04.613 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.083) 0:00:13.131 ******* 2020-03-11T22:28:04.650 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:28:04.652 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.691 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.079) 0:00:13.211 ******* 2020-03-11T22:28:04.731 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:28:04.733 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.777 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.085) 0:00:13.297 ******* 2020-03-11T22:28:04.803 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-11T22:28:04.804 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.851 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.074) 0:00:13.371 ******* 2020-03-11T22:28:04.889 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.926 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.075) 0:00:13.446 ******* 2020-03-11T22:28:04.973 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-11T22:28:04.974 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:05.014 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.087) 0:00:13.534 ******* 2020-03-11T22:28:05.046 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-11T22:28:05.047 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:05.090 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.076) 0:00:13.610 ******* 2020-03-11T22:28:05.139 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-11T22:28:05.140 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:28:05.183 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.092) 0:00:13.703 ******* 2020-03-11T22:28:05.220 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:28:05.259 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.076) 0:00:13.779 ******* 2020-03-11T22:28:05.297 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:28:05.299 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:05.339 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.080) 0:00:13.859 ******* 2020-03-11T22:28:05.605 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-11T22:28:05.607 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:28:05.651 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.311) 0:00:14.171 ******* 2020-03-11T22:28:06.233 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-11T22:28:06.233 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:28:06.709 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:28:06.755 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:01.104) 0:00:15.275 ******* 2020-03-11T22:28:07.282 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi130.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:28:07.761 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:28:07.804 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:01.048) 0:00:16.324 ******* 2020-03-11T22:28:07.843 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:28:07.844 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.882 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.078) 0:00:16.402 ******* 2020-03-11T22:28:07.908 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:28:07.909 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.946 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.064) 0:00:16.467 ******* 2020-03-11T22:28:07.972 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:28:07.974 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.011 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.064) 0:00:16.531 ******* 2020-03-11T22:28:08.037 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:28:08.038 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.076 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.065) 0:00:16.596 ******* 2020-03-11T22:28:08.102 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:28:08.103 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.139 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.062) 0:00:16.659 ******* 2020-03-11T22:28:08.166 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:28:08.167 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.218 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.078) 0:00:16.737 ******* 2020-03-11T22:28:08.244 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-11T22:28:08.246 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.282 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.064) 0:00:16.802 ******* 2020-03-11T22:28:08.308 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:28:08.310 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.345 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.063) 0:00:16.865 ******* 2020-03-11T22:28:08.371 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:28:08.372 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.408 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.062) 0:00:16.928 ******* 2020-03-11T22:28:09.524 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.913953", "end": "2020-03-11 22:28:09.500317", "rc": 0, "start": "2020-03-11 22:28:08.586364", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:09.562 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:01.154) 0:00:18.083 ******* 2020-03-11T22:28:12.815 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:28:12.816 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:28:12.853 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:12 +0000 (0:00:03.290) 0:00:21.373 ******* 2020-03-11T22:28:12.879 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-11T22:28:12.880 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:12.915 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:12 +0000 (0:00:00.061) 0:00:21.435 ******* 2020-03-11T22:28:12.942 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-11T22:28:12.944 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:12.979 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:12 +0000 (0:00:00.063) 0:00:21.499 ******* 2020-03-11T22:28:13.010 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:28:13.012 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:13.060 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:13 +0000 (0:00:00.081) 0:00:21.580 ******* 2020-03-11T22:28:13.087 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:28:13.088 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:13.124 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:13 +0000 (0:00:00.063) 0:00:21.644 ******* 2020-03-11T22:28:13.618 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:28:13.659 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:13 +0000 (0:00:00.534) 0:00:22.179 ******* 2020-03-11T22:28:14.581 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:28:14.583 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:28:14.621 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:14 +0000 (0:00:00.962) 0:00:23.141 ******* 2020-03-11T22:28:15.147 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:28:15.148 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:28:15.624 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:28:16.083 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:28:16.122 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:16 +0000 (0:00:01.500) 0:00:24.642 ******* 2020-03-11T22:28:16.328 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:28:16.329 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:28:16.373 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:16 +0000 (0:00:00.251) 0:00:24.894 ******* 2020-03-11T22:28:16.923 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:28:16.924 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:28:17.390 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:28:17.845 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:28:17.890 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:17 +0000 (0:00:01.516) 0:00:26.410 ******* 2020-03-11T22:28:17.927 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:28:17.928 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:17.964 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:17 +0000 (0:00:00.074) 0:00:26.485 ******* 2020-03-11T22:28:18.001 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:28:18.002 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:28:18.046 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:18 +0000 (0:00:00.081) 0:00:26.566 ******* 2020-03-11T22:28:18.259 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:28:18.261 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:28:18.296 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:18 +0000 (0:00:00.250) 0:00:26.817 ******* 2020-03-11T22:28:18.344 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:28:18.345 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:18.355 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:18.366 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:18.378 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:18.426 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:18 +0000 (0:00:00.128) 0:00:26.945 ******* 2020-03-11T22:28:19.345 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:28:19.346 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:28:19.383 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:19 +0000 (0:00:00.958) 0:00:27.903 ******* 2020-03-11T22:28:19.410 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:19.456 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:19 +0000 (0:00:00.072) 0:00:27.976 ******* 2020-03-11T22:28:19.937 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-11T22:28:19.937 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:28:19.986 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:19 +0000 (0:00:00.529) 0:00:28.506 ******* 2020-03-11T22:28:20.488 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:28:20.489 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:28:20.544 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:20 +0000 (0:00:00.558) 0:00:29.064 ******* 2020-03-11T22:28:20.572 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-11T22:28:20.573 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:20.615 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:20 +0000 (0:00:00.070) 0:00:29.135 ******* 2020-03-11T22:28:20.828 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:28:20.830 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:28:20.865 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:20 +0000 (0:00:00.250) 0:00:29.386 ******* 2020-03-11T22:28:21.131 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:28:21.132 INFO:teuthology.task.ansible.out:fatal: [smithi130.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:28:21.184 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:21 +0000 (0:00:00.318) 0:00:29.704 ******* 2020-03-11T22:28:21.223 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-11T22:28:21.225 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:21.259 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:21 +0000 (0:00:00.075) 0:00:29.780 ******* 2020-03-11T22:28:21.751 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-11T22:28:21.752 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:28:21.794 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:21 +0000 (0:00:00.534) 0:00:30.315 ******* 2020-03-11T22:28:22.596 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:28:22.598 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:27:09 UTC", "ActiveEnterTimestampMonotonic": "11906955", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket nss-lookup.target tmp.mount var-run.mount network.target systemd-tmpfiles-setup.service time-sync.target sysinit.target -.mount basic.target system.slice local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:27:09 UTC", "AssertTimestampMonotonic": "11906127", "Before": "getty@tty1.service plymouth-quit.service xdm.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:27:09 UTC", "ConditionTimestampMonotonic": "11906127", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-11T22:28:22.599 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2387", "ExecMainStartTimestamp": "Wed 2020-03-11 22:27:09 UTC", "ExecMainStartTimestampMonotonic": "11906927", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:27:09 UTC] ; stop_time=[n/a] ; pid=2387 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:27:09 UTC", "InactiveExitTimestampMonotonic": "11906955", "InvocationID": "b1c29d595f5d45d79e9e35b6b72d13a6", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-11T22:28:22.600 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2387", "MemoryAccounting": "yes", "MemoryCurrent": "1376256", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-11T22:28:22.602 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:27:09 UTC", "StateChangeTimestampMonotonic": "11906955", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:27:09 UTC", "WatchdogTimestampMonotonic": "11906954", "WatchdogUSec": "0"}} 2020-03-11T22:28:22.656 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:22 +0000 (0:00:00.862) 0:00:31.177 ******* 2020-03-11T22:28:22.849 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:28:22.851 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001712", "end": "2020-03-11 22:28:22.828960", "rc": 0, "start": "2020-03-11 22:28:22.827248", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:28:22.892 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:22 +0000 (0:00:00.235) 0:00:31.412 ******* 2020-03-11T22:28:23.829 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-11T22:28:23.830 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:28:23.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:23 +0000 (0:00:00.975) 0:00:32.387 ******* 2020-03-11T22:28:24.783 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:28:24.783 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:28:24.823 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.955) 0:00:33.343 ******* 2020-03-11T22:28:26.337 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-11T22:28:26.338 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:28:26.391 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:26 +0000 (0:00:01.568) 0:00:34.911 ******* 2020-03-11T22:28:42.937 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:28:42.941 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.343877", "end": "2020-03-11 22:28:42.906491", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:28:26.562614", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:28:42.978 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:42 +0000 (0:00:16.586) 0:00:51.498 ******* 2020-03-11T22:28:43.456 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:43.457 INFO:teuthology.task.ansible.out:TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:28:43.495 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:43 +0000 (0:00:00.516) 0:00:52.015 ******* 2020-03-11T22:28:43.699 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:28:43.700 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.010599", "end": "2020-03-11 22:28:43.675298", "rc": 0, "start": "2020-03-11 22:28:43.664699", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:43.740 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:43 +0000 (0:00:00.244) 0:00:52.260 ******* 2020-03-11T22:28:43.966 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:28:43.966 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.020367", "end": "2020-03-11 22:28:43.934972", "rc": 0, "start": "2020-03-11 22:28:43.914605", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:44.006 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:44 +0000 (0:00:00.266) 0:00:52.527 ******* 2020-03-11T22:29:00.364 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:29:00.366 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.154809", "end": "2020-03-11 22:29:00.334816", "rc": 0, "start": "2020-03-11 22:28:44.180007", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:00.403 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:00 +0000 (0:00:16.396) 0:01:08.924 ******* 2020-03-11T22:29:00.604 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:29:00.606 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:29:00.644 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:00 +0000 (0:00:00.240) 0:01:09.164 ******* 2020-03-11T22:29:00.679 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:29:00.680 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:29:00.719 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:00 +0000 (0:00:00.074) 0:01:09.239 ******* 2020-03-11T22:29:00.764 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:29:00.765 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:29:00.803 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:00 +0000 (0:00:00.084) 0:01:09.323 ******* 2020-03-11T22:29:00.842 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:29:00.843 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:29:00.885 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:00 +0000 (0:00:00.082) 0:01:09.406 ******* 2020-03-11T22:29:00.922 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:29:00.923 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:29:00.964 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:00 +0000 (0:00:00.078) 0:01:09.484 ******* 2020-03-11T22:29:01.156 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:29:01.158 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:29:01.196 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:01 +0000 (0:00:00.231) 0:01:09.716 ******* 2020-03-11T22:29:01.387 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:29:01.389 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:29:01.425 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:01 +0000 (0:00:00.229) 0:01:09.945 ******* 2020-03-11T22:29:01.457 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-11T22:29:01.458 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:01.498 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:01 +0000 (0:00:00.073) 0:01:10.018 ******* 2020-03-11T22:29:01.524 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:29:01.525 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:01.564 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:01 +0000 (0:00:00.065) 0:01:10.084 ******* 2020-03-11T22:29:01.785 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:29:01.786 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:29:01.822 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:01 +0000 (0:00:00.258) 0:01:10.342 ******* 2020-03-11T22:29:02.383 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:29:02.385 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:29:02.424 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:02 +0000 (0:00:00.601) 0:01:10.944 ******* 2020-03-11T22:29:02.460 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-11T22:29:02.462 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi130"}, "changed": false} 2020-03-11T22:29:02.499 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:02 +0000 (0:00:00.074) 0:01:11.019 ******* 2020-03-11T22:29:03.527 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-11T22:29:03.529 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi130.front.sepia.ceph.com", "ansible_hostname": "smithi130", "ansible_nodename": "smithi130"}, "changed": true, "name": "smithi130"} 2020-03-11T22:29:03.566 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:03 +0000 (0:00:01.067) 0:01:12.086 ******* 2020-03-11T22:29:04.051 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-11T22:29:04.052 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:29:04.089 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:04 +0000 (0:00:00.522) 0:01:12.609 ******* 2020-03-11T22:29:04.562 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:29:04.564 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:29:04.601 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:04 +0000 (0:00:00.511) 0:01:13.121 ******* 2020-03-11T22:29:05.259 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-11T22:29:05.260 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:29:05.296 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:05 +0000 (0:00:00.695) 0:01:13.816 ******* 2020-03-11T22:29:05.784 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-11T22:29:05.786 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:29:05.824 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:05 +0000 (0:00:00.528) 0:01:14.345 ******* 2020-03-11T22:29:06.026 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:29:06.026 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:29:06.063 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:06 +0000 (0:00:00.238) 0:01:14.583 ******* 2020-03-11T22:29:06.265 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:29:06.266 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:29:06.304 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:06 +0000 (0:00:00.241) 0:01:14.825 ******* 2020-03-11T22:29:06.531 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:29:06.532 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:29:06.589 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:06 +0000 (0:00:00.284) 0:01:15.109 ******* 2020-03-11T22:29:07.081 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:29:07.082 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:29:07.130 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:07 +0000 (0:00:00.533) 0:01:15.642 ******* 2020-03-11T22:29:07.603 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:29:07.604 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:29:07.645 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:07 +0000 (0:00:00.523) 0:01:16.165 ******* 2020-03-11T22:29:09.445 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:29:09.447 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.594063", "end": "2020-03-11 22:29:09.414174", "rc": 0, "start": "2020-03-11 22:29:07.820111", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:09.509 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:09 +0000 (0:00:01.863) 0:01:18.029 ******* 2020-03-11T22:29:10.012 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:29:10.012 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:29:10.478 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:29:10.942 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:29:10.990 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:10 +0000 (0:00:01.481) 0:01:19.510 ******* 2020-03-11T22:29:11.502 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-11T22:29:11.504 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:29:11.542 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:11 +0000 (0:00:00.552) 0:01:20.063 ******* 2020-03-11T22:29:12.035 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:29:12.036 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:29:12.481 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:29:12.999 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:29:13.048 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:01.505) 0:01:21.568 ******* 2020-03-11T22:29:13.565 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-11T22:29:13.567 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.303092", "end": "2020-03-11 22:29:13.534573", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:29:13.231481", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-11T22:29:13.571 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:29:13.635 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.586) 0:01:22.155 ******* 2020-03-11T22:29:13.660 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-11T22:29:13.662 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.700 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.065) 0:01:22.220 ******* 2020-03-11T22:29:13.747 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-11T22:29:13.748 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.793 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.092) 0:01:22.313 ******* 2020-03-11T22:29:13.820 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-11T22:29:13.821 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.873 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.080) 0:01:22.393 ******* 2020-03-11T22:29:13.912 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:13.914 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-11T22:29:13.951 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.077) 0:01:22.471 ******* 2020-03-11T22:29:13.990 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-11T22:29:13.992 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.039 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.087) 0:01:22.559 ******* 2020-03-11T22:29:14.077 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-11T22:29:14.078 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.113 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.073) 0:01:22.633 ******* 2020-03-11T22:29:14.317 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-11T22:29:14.319 INFO:teuthology.task.ansible.out:fatal: [smithi130.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006274", "end": "2020-03-11 22:29:14.297037", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:29:14.290763", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-11T22:29:14.381 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.268) 0:01:22.901 ******* 2020-03-11T22:29:14.409 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-11T22:29:14.411 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.448 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.066) 0:01:22.968 ******* 2020-03-11T22:29:17.026 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:29:17.027 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:29:17.072 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:02.623) 0:01:25.592 ******* 2020-03-11T22:29:20.043 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:29:20.044 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:29:20.081 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:03.009) 0:01:28.601 ******* 2020-03-11T22:29:25.023 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-11T22:29:25.025 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:04.717881", "end": "2020-03-11 22:29:24.994697", "rc": 0, "start": "2020-03-11 22:29:20.276816", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\""], "stdout": "CentOS-8 - AppStream 140 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 130 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 50 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 32 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 168 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:29:20 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-11T22:29:25.026 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 140 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 130 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 50 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 32 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 168 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:29:20 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:29:25.083 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:05.002) 0:01:33.603 ******* 2020-03-11T22:29:32.558 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:32.560 INFO:teuthology.task.ansible.out:TASK [testnode : Install packages] ********************************************* changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:29:32.602 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:07.518) 0:01:41.122 ******* 2020-03-11T22:29:32.628 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:29:32.630 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.667 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.064) 0:01:41.187 ******* 2020-03-11T22:29:33.621 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:29:33.623 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:29:33.671 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:01.004) 0:01:42.191 ******* 2020-03-11T22:29:33.718 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:33.719 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.756 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.084) 0:01:42.276 ******* 2020-03-11T22:29:34.090 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-11T22:29:34.092 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket system.slice basic.target polkit.service dbus.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service iptables.service ebtables.service ip6tables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-11T22:29:34.098 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", 2020-03-11T22:29:34.101 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-11T22:29:34.102 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:29:34.129 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:34 +0000 (0:00:00.373) 0:01:42.649 ******* 2020-03-11T22:29:34.424 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-11T22:29:34.429 INFO:teuthology.task.ansible.out:fatal: [smithi130.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:29:34.474 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:34 +0000 (0:00:00.344) 0:01:42.994 ******* 2020-03-11T22:29:35.199 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:29:35.200 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:29:35.242 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.768) 0:01:43.762 ******* 2020-03-11T22:29:35.268 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:29:35.270 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.310 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.068) 0:01:43.831 ******* 2020-03-11T22:29:35.337 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:29:35.376 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.065) 0:01:43.896 ******* 2020-03-11T22:29:35.404 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:29:35.405 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.441 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.065) 0:01:43.961 ******* 2020-03-11T22:29:35.467 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-11T22:29:35.468 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.513 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.071) 0:01:44.033 ******* 2020-03-11T22:29:35.551 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-11T22:29:35.553 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.563 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.607 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.093) 0:01:44.127 ******* 2020-03-11T22:29:35.633 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-11T22:29:35.635 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.671 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.064) 0:01:44.192 ******* 2020-03-11T22:29:35.698 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:29:35.739 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.067) 0:01:44.259 ******* 2020-03-11T22:29:35.765 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:29:35.767 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.804 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.064) 0:01:44.324 ******* 2020-03-11T22:29:35.840 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:29:35.842 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.881 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.077) 0:01:44.401 ******* 2020-03-11T22:29:35.907 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:29:35.909 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:35.944 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.062) 0:01:44.464 ******* 2020-03-11T22:29:35.970 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:35.971 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.009 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.064) 0:01:44.529 ******* 2020-03-11T22:29:36.035 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.073 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.064) 0:01:44.593 ******* 2020-03-11T22:29:36.100 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:29:36.102 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.140 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.067) 0:01:44.660 ******* 2020-03-11T22:29:36.168 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:29:36.168 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.215 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.074) 0:01:44.735 ******* 2020-03-11T22:29:36.241 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-11T22:29:36.242 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.289 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.074) 0:01:44.809 ******* 2020-03-11T22:29:36.315 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:29:36.316 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.360 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.071) 0:01:44.881 ******* 2020-03-11T22:29:36.387 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:29:36.389 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.429 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.068) 0:01:44.949 ******* 2020-03-11T22:29:36.455 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:29:36.456 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.497 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.067) 0:01:45.017 ******* 2020-03-11T22:29:36.524 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-11T22:29:36.525 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.568 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.071) 0:01:45.088 ******* 2020-03-11T22:29:36.600 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:29:36.601 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.642 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.074) 0:01:45.163 ******* 2020-03-11T22:29:36.689 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:29:36.690 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.741 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.098) 0:01:45.261 ******* 2020-03-11T22:29:36.768 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:29:36.769 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.815 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.074) 0:01:45.335 ******* 2020-03-11T22:29:36.851 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:29:36.853 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.891 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.075) 0:01:45.411 ******* 2020-03-11T22:29:36.918 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:29:36.918 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:36.963 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.072) 0:01:45.483 ******* 2020-03-11T22:29:37.001 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:29:37.003 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.053 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.090) 0:01:45.573 ******* 2020-03-11T22:29:37.080 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:29:37.081 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.120 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.067) 0:01:45.640 ******* 2020-03-11T22:29:37.146 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:29:37.148 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.189 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.068) 0:01:45.709 ******* 2020-03-11T22:29:37.216 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:29:37.218 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.254 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.065) 0:01:45.775 ******* 2020-03-11T22:29:37.280 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:29:37.282 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.318 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.063) 0:01:45.838 ******* 2020-03-11T22:29:37.344 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:29:37.345 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.384 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.065) 0:01:45.904 ******* 2020-03-11T22:29:37.411 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:29:37.413 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.459 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.075) 0:01:45.979 ******* 2020-03-11T22:29:37.486 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:29:37.487 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.538 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.078) 0:01:46.058 ******* 2020-03-11T22:29:37.575 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:29:37.577 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:37.624 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.086) 0:01:46.144 ******* 2020-03-11T22:29:37.848 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:37.850 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:29:37.885 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:37 +0000 (0:00:00.260) 0:01:46.405 ******* 2020-03-11T22:29:38.099 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:38.100 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:38.152 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.266) 0:01:46.672 ******* 2020-03-11T22:29:38.357 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:38.358 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:29:38.410 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.258) 0:01:46.930 ******* 2020-03-11T22:29:38.441 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:29:38.442 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.490 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.080) 0:01:47.011 ******* 2020-03-11T22:29:38.519 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:29:38.520 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.578 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.087) 0:01:47.098 ******* 2020-03-11T22:29:38.620 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:29:38.620 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.681 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.103) 0:01:47.202 ******* 2020-03-11T22:29:38.714 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:38.715 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.754 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.072) 0:01:47.274 ******* 2020-03-11T22:29:38.781 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:38.782 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.843 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.088) 0:01:47.363 ******* 2020-03-11T22:29:38.869 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:38.871 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.910 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.067) 0:01:47.431 ******* 2020-03-11T22:29:38.936 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:29:38.937 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:38.975 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:00.064) 0:01:47.495 ******* 2020-03-11T22:29:39.000 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:29:39.002 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.046 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.066) 0:01:47.562 ******* 2020-03-11T22:29:39.079 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:29:39.080 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.117 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.075) 0:01:47.638 ******* 2020-03-11T22:29:39.159 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-11T22:29:39.160 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.196 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.078) 0:01:47.716 ******* 2020-03-11T22:29:39.222 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:29:39.223 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.275 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.078) 0:01:47.795 ******* 2020-03-11T22:29:39.302 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:29:39.306 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.340 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.065) 0:01:47.861 ******* 2020-03-11T22:29:39.377 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:29:39.378 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.415 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.074) 0:01:47.935 ******* 2020-03-11T22:29:39.441 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:29:39.442 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.484 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.069) 0:01:48.004 ******* 2020-03-11T22:29:39.511 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:29:39.512 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.560 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.075) 0:01:48.080 ******* 2020-03-11T22:29:39.596 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-11T22:29:39.598 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.636 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.075) 0:01:48.156 ******* 2020-03-11T22:29:39.663 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-11T22:29:39.665 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.712 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.076) 0:01:48.233 ******* 2020-03-11T22:29:39.738 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:29:39.740 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.784 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.071) 0:01:48.304 ******* 2020-03-11T22:29:39.811 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.859 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.075) 0:01:48.380 ******* 2020-03-11T22:29:39.886 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:29:39.888 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.922 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.062) 0:01:48.442 ******* 2020-03-11T22:29:39.948 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-11T22:29:39.950 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:39.986 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:39 +0000 (0:00:00.064) 0:01:48.506 ******* 2020-03-11T22:29:40.018 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-11T22:29:40.020 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.055 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.068) 0:01:48.575 ******* 2020-03-11T22:29:40.091 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:40.092 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:29:40.128 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.073) 0:01:48.648 ******* 2020-03-11T22:29:40.174 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:29:40.176 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.185 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:29:40.196 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.197 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.208 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.219 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.229 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.271 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.142) 0:01:48.791 ******* 2020-03-11T22:29:40.493 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-11T22:29:40.494 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.018544", "end": "2020-03-11 22:29:40.463680", "rc": 0, "start": "2020-03-11 22:29:40.445136", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:29:40.536 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.265) 0:01:49.056 ******* 2020-03-11T22:29:40.573 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:29:40.575 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.634 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.098) 0:01:49.155 ******* 2020-03-11T22:29:40.673 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:29:40.675 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.728 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.093) 0:01:49.248 ******* 2020-03-11T22:29:40.754 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:29:40.756 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.793 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.064) 0:01:49.313 ******* 2020-03-11T22:29:40.819 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-11T22:29:40.820 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:40.858 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:40 +0000 (0:00:00.065) 0:01:49.378 ******* 2020-03-11T22:29:41.814 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:29:41.815 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:29:41.857 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:41 +0000 (0:00:00.999) 0:01:50.377 ******* 2020-03-11T22:29:41.911 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:29:41.912 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:29:41.971 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:41 +0000 (0:00:00.114) 0:01:50.492 ******* 2020-03-11T22:29:42.175 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:29:42.176 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005195", "end": "2020-03-11 22:29:42.150103", "rc": 0, "start": "2020-03-11 22:29:42.144908", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:42.216 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.244) 0:01:50.737 ******* 2020-03-11T22:29:42.388 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:29:42.388 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:42.450 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.233) 0:01:50.970 ******* 2020-03-11T22:29:42.479 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:29:42.516 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:42 +0000 (0:00:00.065) 0:01:51.036 ******* 2020-03-11T22:29:42.689 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:29:42.690 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XK59', u'wwn-0x5000c5009293fead'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XK59', u'wwn': u'0x5000c5009293fead', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XK59-part1', u'wwn-0x5000c5009293fead-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK59", "wwn-0x5000c5009293fead"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK59-part1", "wwn-0x5000c5009293fead-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XK59", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:29:42.691 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009293fead"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:44.279 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-YvK0oc-9x8r-3xzF-sp7m-LM9V-12rc-VdBCaA', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN', u'nvme-nvme.8086-50484654363230353030513134303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.400281", "end": "2020-03-11 22:29:44.246144", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-YvK0oc-9x8r-3xzF-sp7m-LM9V-12rc-VdBCaA", "nvme-INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN", "nvme-nvme.8086-50484654363230353030513134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:42.845863", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-11T22:29:44.280 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:45.520 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgXhc7XPd8SyBrxI7h6b55LOFO58Qorsk3'], u'uuids': [u'19fa0b3f-33e2-41c0-ab4c-a616c0449eec']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.044108", "end": "2020-03-11 22:29:45.486161", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgXhc7XPd8SyBrxI7h6b55LOFO58Qorsk3"], "labels": [], "masters": [], "uuids": ["19fa0b3f-33e2-41c0-ab4c-a616c0449eec"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:44.442053", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:46.771 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgLfGpLQhwN1nfJ6vko9p01KX9RYHqmi8V'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.040463", "end": "2020-03-11 22:29:46.737045", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgLfGpLQhwN1nfJ6vko9p01KX9RYHqmi8V"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:45.696582", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:48.024 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgWVn0lMZ87e10dd9a2hybkpI6cfAMUNVq'], u'uuids': [u'1997d474-d208-4b1c-b0df-e1c4c0af2900']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.041327", "end": "2020-03-11 22:29:47.979786", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgWVn0lMZ87e10dd9a2hybkpI6cfAMUNVq"], "labels": [], "masters": [], "uuids": ["1997d474-d208-4b1c-b0df-e1c4c0af2900"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:46.938459", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:49.271 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Egb17unLW4bYhSLQCqR0E0067jIeF0RXNa'], u'uuids': [u'a267beb8-42c6-4cb7-ac43-c0bbe6e29591']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.040314", "end": "2020-03-11 22:29:49.229313", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Egb17unLW4bYhSLQCqR0E0067jIeF0RXNa"], "labels": [], "masters": [], "uuids": ["a267beb8-42c6-4cb7-ac43-c0bbe6e29591"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:48.188999", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:50.613 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Eg3lFvbNhFIA7V31AfNJQowSdIW4gT5mcr'], u'uuids': [u'095ede75-e85a-4e8d-8152-564a4f22c069']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.040062", "end": "2020-03-11 22:29:50.479355", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Eg3lFvbNhFIA7V31AfNJQowSdIW4gT5mcr"], "labels": [], "masters": [], "uuids": ["095ede75-e85a-4e8d-8152-564a4f22c069"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:49.439293", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:29:50.662 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:50 +0000 (0:00:08.146) 0:01:59.182 ******* 2020-03-11T22:29:50.709 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-11T22:29:50.710 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XK59', u'wwn-0x5000c5009293fead'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XK59', u'wwn': u'0x5000c5009293fead', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XK59-part1', u'wwn-0x5000c5009293fead-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK59", "wwn-0x5000c5009293fead"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK59-part1", "wwn-0x5000c5009293fead-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XK59", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:29:50.712 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009293fead"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:50.940 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-YvK0oc-9x8r-3xzF-sp7m-LM9V-12rc-VdBCaA', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN', u'nvme-nvme.8086-50484654363230353030513134303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037284", "end": "2020-03-11 22:29:50.909521", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-YvK0oc-9x8r-3xzF-sp7m-LM9V-12rc-VdBCaA", "nvme-INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN", "nvme-nvme.8086-50484654363230353030513134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:50.872237", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-11T22:29:50.942 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0345879 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0345879 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:51.275 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgXhc7XPd8SyBrxI7h6b55LOFO58Qorsk3'], u'uuids': [u'19fa0b3f-33e2-41c0-ab4c-a616c0449eec']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.137813", "end": "2020-03-11 22:29:51.244408", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgXhc7XPd8SyBrxI7h6b55LOFO58Qorsk3"], "labels": [], "masters": [], "uuids": ["19fa0b3f-33e2-41c0-ab4c-a616c0449eec"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:51.106595", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135173 s, 853 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135173 s, 853 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:51.609 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgLfGpLQhwN1nfJ6vko9p01KX9RYHqmi8V'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.138349", "end": "2020-03-11 22:29:51.580289", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgLfGpLQhwN1nfJ6vko9p01KX9RYHqmi8V"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": 2020-03-11T22:29:51.610 INFO:teuthology.task.ansible.out:"0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:51.441940", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135757 s, 850 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135757 s, 850 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:51.938 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgWVn0lMZ87e10dd9a2hybkpI6cfAMUNVq'], u'uuids': [u'1997d474-d208-4b1c-b0df-e1c4c0af2900']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.137606", "end": "2020-03-11 22:29:51.905429", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44EgWVn0lMZ87e10dd9a2hybkpI6cfAMUNVq"], "labels": [], "masters": [], "uuids": ["1997d474-d208-4b1c-b0df-e1c4c0af2900"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:51.767823", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.134907 s, 855 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.134907 s, 855 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:52.268 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Egb17unLW4bYhSLQCqR0E0067jIeF0RXNa'], u'uuids': [u'a267beb8-42c6-4cb7-ac43-c0bbe6e29591']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144140", "end": "2020-03-11 22:29:52.241226", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Egb17unLW4bYhSLQCqR0E0067jIeF0RXNa"], "labels": [], "masters": [], "uuids": ["a267beb8-42c6-4cb7-ac43-c0bbe6e29591"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:52.097086", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141371 s, 816 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141371 s, 816 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:52.590 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Eg3lFvbNhFIA7V31AfNJQowSdIW4gT5mcr'], u'uuids': [u'095ede75-e85a-4e8d-8152-564a4f22c069']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.137163", "end": "2020-03-11 22:29:52.561319", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-vuIbI31xtcPoDfo21pZFY2AdEfUa44Eg3lFvbNhFIA7V31AfNJQowSdIW4gT5mcr"], "labels": [], "masters": [], "uuids": ["095ede75-e85a-4e8d-8152-564a4 2020-03-11T22:29:52.591 INFO:teuthology.task.ansible.out:f22c069"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:29:52.424156", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.134611 s, 857 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.134611 s, 857 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:52.636 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:52 +0000 (0:00:01.973) 0:02:01.156 ******* 2020-03-11T22:29:52.911 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:29:52.912 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.073725", "end": "2020-03-11 22:29:52.877282", "rc": 0, "start": "2020-03-11 22:29:52.803557", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:52.954 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:52 +0000 (0:00:00.318) 0:02:01.474 ******* 2020-03-11T22:29:53.199 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-11T22:29:53.201 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045655", "end": "2020-03-11 22:29:53.171857", "rc": 0, "start": "2020-03-11 22:29:53.126202", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:53.238 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.283) 0:02:01.758 ******* 2020-03-11T22:29:53.263 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:29:53.303 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.065) 0:02:01.823 ******* 2020-03-11T22:29:53.329 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:29:53.331 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:53.369 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.065) 0:02:01.889 ******* 2020-03-11T22:29:53.395 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:29:53.396 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:53.432 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.063) 0:02:01.953 ******* 2020-03-11T22:29:53.458 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:29:53.460 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:53.498 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.065) 0:02:02.018 ******* 2020-03-11T22:29:53.525 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:29:53.526 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:53.565 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.064) 0:02:02.083 ******* 2020-03-11T22:29:53.767 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:29:53.768 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:53.806 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.242) 0:02:02.326 ******* 2020-03-11T22:29:53.853 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:29:53.854 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:53.903 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.097) 0:02:02.423 ******* 2020-03-11T22:29:53.940 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-11T22:29:53.941 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:53.978 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:53 +0000 (0:00:00.075) 0:02:02.499 ******* 2020-03-11T22:29:54.015 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:54.016 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:54.053 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:54 +0000 (0:00:00.074) 0:02:02.573 ******* 2020-03-11T22:29:54.089 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:29:54.091 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:54.130 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:54 +0000 (0:00:00.077) 0:02:02.650 ******* 2020-03-11T22:29:54.167 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:29:54.168 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:54.227 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:54 +0000 (0:00:00.097) 0:02:02.747 ******* 2020-03-11T22:29:54.256 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-11T22:29:54.258 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:54.304 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:54 +0000 (0:00:00.076) 0:02:02.824 ******* 2020-03-11T22:29:54.341 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-11T22:29:54.342 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:54.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:54 +0000 (0:00:00.075) 0:02:02.900 ******* 2020-03-11T22:29:54.971 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:29:55.008 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:55 +0000 (0:00:00.627) 0:02:03.528 ******* 2020-03-11T22:29:55.572 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:29:55.573 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:55.908 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:56.241 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:56.585 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:56.938 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:29:56.978 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:56 +0000 (0:00:01.969) 0:02:05.498 ******* 2020-03-11T22:29:57.181 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:29:57.183 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:29:57.226 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:57 +0000 (0:00:00.247) 0:02:05.746 ******* 2020-03-11T22:29:57.286 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:29:57.286 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:57.474 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:57.654 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:57.843 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:58.031 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:29:58.073 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:58 +0000 (0:00:00.847) 0:02:06.593 ******* 2020-03-11T22:29:58.286 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:29:58.287 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:58.330 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:58 +0000 (0:00:00.257) 0:02:06.851 ******* 2020-03-11T22:29:58.368 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-11T22:29:58.369 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:29:58.413 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:58 +0000 (0:00:00.082) 0:02:06.933 ******* 2020-03-11T22:29:59.878 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:29:59.880 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:29:59.918 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:59 +0000 (0:00:01.504) 0:02:08.438 ******* 2020-03-11T22:30:00.402 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:30:00.403 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:30:00.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.521) 0:02:08.960 ******* 2020-03-11T22:30:00.465 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-11T22:30:00.467 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:00.503 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.063) 0:02:09.023 ******* 2020-03-11T22:30:00.530 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:30:00.531 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:00.574 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.071) 0:02:09.094 ******* 2020-03-11T22:30:00.610 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-11T22:30:00.611 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:00.649 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.075) 0:02:09.169 ******* 2020-03-11T22:30:00.685 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:30:00.687 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:00.743 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.093) 0:02:09.263 ******* 2020-03-11T22:30:00.781 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-11T22:30:00.782 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:00.823 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.079) 0:02:09.343 ******* 2020-03-11T22:30:00.849 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:30:00.850 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:00.886 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.063) 0:02:09.407 ******* 2020-03-11T22:30:00.912 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:30:00.914 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:00.951 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:00 +0000 (0:00:00.064) 0:02:09.471 ******* 2020-03-11T22:30:00.976 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:30:00.978 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:01.015 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:01 +0000 (0:00:00.064) 0:02:09.535 ******* 2020-03-11T22:30:01.041 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:30:01.042 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:01.078 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:01 +0000 (0:00:00.063) 0:02:09.598 ******* 2020-03-11T22:30:01.104 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:30:01.105 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:01.143 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:01 +0000 (0:00:00.064) 0:02:09.663 ******* 2020-03-11T22:30:01.170 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-11T22:30:01.171 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:01.208 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:01 +0000 (0:00:00.065) 0:02:09.728 ******* 2020-03-11T22:30:01.704 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:30:01.752 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:01 +0000 (0:00:00.544) 0:02:10.272 ******* 2020-03-11T22:30:02.067 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:30:02.069 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:27:09 UTC", "ActiveEnterTimestampMonotonic": "11853992", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount ntpd.service systemd-journald.socket sntp.service systemd-tmpfiles-setup.service basic.target ntpdate.service sysinit.target -.mount system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:27:07 UTC", "AssertTimestampMonotonic": "10286182", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:27:07 UTC", "ConditionTimestampMonotonic": "10286010", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-11T22:30:02.071 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2317", "ExecMainStartTimestamp": "Wed 2020-03-11 22:27:08 UTC", "ExecMainStartTimestampMonotonic": "10389094", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:27:07 UTC", "InactiveExitTimestampMonotonic": "10296929", "InvocationID": "cd0194b65623474a9ab524be58b66a48", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-11T22:30:02.072 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2317", "MemoryAccounting": "yes", "MemoryCurrent": "2789376", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-11T22:30:02.074 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:27:09 UTC", "StateChangeTimestampMonotonic": "11853992", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:27:08 UTC", "WatchdogTimestampMonotonic": "10389103", "WatchdogUSec": "0"}} 2020-03-11T22:30:02.109 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:02 +0000 (0:00:00.356) 0:02:10.629 ******* 2020-03-11T22:30:02.135 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-11T22:30:02.136 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:02.173 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:02 +0000 (0:00:00.063) 0:02:10.693 ******* 2020-03-11T22:30:02.200 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-11T22:30:02.200 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:02.238 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:02 +0000 (0:00:00.064) 0:02:10.758 ******* 2020-03-11T22:30:02.265 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:02.301 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:02 +0000 (0:00:00.062) 0:02:10.821 ******* 2020-03-11T22:30:02.503 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:30:02.556 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:02 +0000 (0:00:00.255) 0:02:11.076 ******* 2020-03-11T22:30:03.031 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:30:03.069 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:03 +0000 (0:00:00.512) 0:02:11.589 ******* 2020-03-11T22:30:04.230 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:30:04.230 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:30:04.270 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:04 +0000 (0:00:01.201) 0:02:12.791 ******* 2020-03-11T22:30:04.297 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:30:04.299 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:04.352 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:04 +0000 (0:00:00.081) 0:02:12.872 ******* 2020-03-11T22:30:04.378 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-11T22:30:04.379 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-11T22:30:04.418 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:04 +0000 (0:00:00.065) 0:02:12.938 ******* 2020-03-11T22:30:04.610 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:30:04.611 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:30:04.667 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:04 +0000 (0:00:00.249) 0:02:13.187 ******* 2020-03-11T22:30:04.694 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:30:04.696 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:04.732 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:04 +0000 (0:00:00.064) 0:02:13.252 ******* 2020-03-11T22:30:04.768 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:30:04.769 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:30:04.805 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:04 +0000 (0:00:00.073) 0:02:13.325 ******* 2020-03-11T22:30:04.833 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:04.870 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:04 +0000 (0:00:00.065) 0:02:13.390 ******* 2020-03-11T22:30:05.071 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi130.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:30:05.113 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.242) 0:02:13.633 ******* 2020-03-11T22:30:05.306 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-11T22:30:05.307 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:30:05.348 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.234) 0:02:13.868 ******* 2020-03-11T22:30:05.374 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-11T22:30:05.375 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.418 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.069) 0:02:13.938 ******* 2020-03-11T22:30:05.455 INFO:teuthology.task.ansible.out: 2020-03-11T22:30:05.457 INFO:teuthology.task.ansible.out:TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi130.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi130.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.468 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.470 INFO:teuthology.task.ansible.out: skipping: [smithi130.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.480 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.482 INFO:teuthology.task.ansible.out: 2020-03-11T22:30:05.526 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.107) 0:02:14.046 ******* 2020-03-11T22:30:05.563 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:30:05.564 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.566 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.568 INFO:teuthology.task.ansible.out: 2020-03-11T22:30:05.579 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.581 INFO:teuthology.task.ansible.out: skipping: [smithi130.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.591 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.592 INFO:teuthology.task.ansible.out: 2020-03-11T22:30:05.638 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.112) 0:02:14.158 ******* 2020-03-11T22:30:05.664 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:30:05.665 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.706 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.067) 0:02:14.226 ******* 2020-03-11T22:30:05.742 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:30:05.751 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.782 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.076) 0:02:14.302 ******* 2020-03-11T22:30:05.818 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:30:05.820 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.863 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.080) 0:02:14.383 ******* 2020-03-11T22:30:05.890 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-11T22:30:05.892 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.929 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.066) 0:02:14.449 ******* 2020-03-11T22:30:05.955 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:30:05.957 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:05.996 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:05 +0000 (0:00:00.066) 0:02:14.516 ******* 2020-03-11T22:30:06.022 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:30:06.023 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:06.063 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:06 +0000 (0:00:00.067) 0:02:14.583 ******* 2020-03-11T22:30:06.099 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:30:06.100 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:06.143 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:06 +0000 (0:00:00.080) 0:02:14.664 ******* 2020-03-11T22:30:06.180 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:30:06.181 INFO:teuthology.task.ansible.out:skipping: [smithi130.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:06.226 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:06 +0000 (0:00:00.082) 0:02:14.747 ******* 2020-03-11T22:30:06.491 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:30:06.493 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005120", "end": "2020-03-11 22:30:06.455051", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:30:06.449931", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:06.709 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002878", "end": "2020-03-11 22:30:06.672714", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:30:06.669836", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:06.898 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002825", "end": "2020-03-11 22:30:06.856844", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:30:06.854019", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:07.105 INFO:teuthology.task.ansible.out:ok: [smithi130.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.021766", "end": "2020-03-11 22:30:07.063539", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:30:07.041773", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:07.151 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:07 +0000 (0:00:00.924) 0:02:15.671 ******* 2020-03-11T22:30:07.176 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:30:07.216 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:07 +0000 (0:00:00.065) 0:02:15.736 ******* 2020-03-11T22:30:07.411 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-11T22:30:07.413 INFO:teuthology.task.ansible.out:changed: [smithi130.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:30:07.514 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2020-03-11T22:30:07.515 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:30:07.518 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-03-11T22:30:07.520 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:30:07.522 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-03-11T22:30:07.522 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:30:07.523 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi130.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Wednesday 11 March 2020 22:30:07 +0000 (0:00:00.306) 0:02:16.043 ******* =============================================================================== 2020-03-11T22:30:07.525 INFO:teuthology.task.ansible.out:common : nrpe - Remove SELinux policy package -------------------------- 16.59s common : nrpe - Load SELinux policy package ---------------------------- 16.40s testnode : Zap all non-root disks --------------------------------------- 8.15s testnode : Install packages --------------------------------------------- 7.52s testnode : Base group workaround for BZ1782899 -------------------------- 5.00s common : Install Kerberos Packages (RedHat) ----------------------------- 3.29s testnode : Ensure ceph dependency packages are not present. ------------- 3.01s Gathering Facts --------------------------------------------------------- 2.73s testnode : Ensure ceph packages are not present. ------------------------ 2.62s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.97s testnode : Create logical volume(s) ------------------------------------- 1.97s testnode : Rebuild rpmdb ------------------------------------------------ 1.86s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.57s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.52s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.51s testnode : Create xfs filesystem on /dev/vg_nvme/lv_5 ------------------- 1.50s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.50s testnode : Configure local mirrorlists ---------------------------------- 1.48s testnode : Install packages via pip ------------------------------------- 1.20s common : Set the hardware clock ----------------------------------------- 1.15s 2020-03-11T22:30:07.855 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2020-03-11T22:30:07.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:07.858 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi130.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:07.926 INFO:teuthology.orchestra.run.smithi130:> true 2020-03-11T22:30:08.018 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:30:08.035 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:30:08.036 INFO:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:30:08.094 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:30:08.112 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:30:08.154 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2020-03-11T22:30:08.168 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2020-03-11T22:30:08.186 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:30:08.204 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:30:08.265 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2020-03-11T22:30:08.268 INFO:teuthology.orchestra.run.smithi130.stdout:210 Number of sources = 4 2020-03-11T22:30:08.268 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:30:08.268 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2020-03-11T22:30:08.268 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:30:08.268 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:30:08.269 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:30:08.269 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:30:08.269 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:30:08.279 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:30:08.279 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:30:08.279 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:30:08.279 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:30:08.288 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:30:08.288 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:30:08.289 INFO:teuthology.orchestra.run.smithi130:> sudo lsb_release -is 2020-03-11T22:30:08.345 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS 2020-03-11T22:30:08.346 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:30:08.347 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:30:08.347 INFO:teuthology.orchestra.run.smithi130:> sudo lsb_release -is 2020-03-11T22:30:08.421 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS 2020-03-11T22:30:08.423 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:30:08.423 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:30:08.644 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:30:08.645 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:30:08.825 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:30:08.825 INFO:teuthology.orchestra.run.smithi130:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:30:08.871 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:30:08.871 INFO:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:30:08.940 INFO:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:30:09.020 INFO:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:30:09.087 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2020-03-11T22:30:09.088 INFO:teuthology.orchestra.run.smithi130:> sudo yum clean all 2020-03-11T22:30:09.440 INFO:teuthology.orchestra.run.smithi130.stdout:239 files removed 2020-03-11T22:30:09.470 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw 2020-03-11T22:30:10.164 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 195 kB/s | 69 kB 00:00 2020-03-11T22:30:10.463 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 50 kB/s | 14 kB 00:00 2020-03-11T22:30:10.772 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 4.9 kB/s | 1.4 kB 00:00 2020-03-11T22:30:10.926 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-8 - AppStream 48 MB/s | 6.5 MB 00:00 2020-03-11T22:30:12.087 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-8 - Base 38 MB/s | 5.0 MB 00:00 2020-03-11T22:30:12.866 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-8 - Extras 33 kB/s | 2.1 kB 00:00 2020-03-11T22:30:13.029 INFO:teuthology.orchestra.run.smithi130.stdout:Copr repo for ceph-el8 owned by ktdreyer 322 kB/s | 38 kB 00:00 2020-03-11T22:30:13.404 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 18 MB/s | 6.0 MB 00:00 2020-03-11T22:30:14.627 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 5.4 MB/s | 30 kB 00:00 2020-03-11T22:30:15.591 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:30:15.594 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:30:15.594 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2020-03-11T22:30:15.594 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:30:15.594 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:30:15.594 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:30:15.595 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2020-03-11T22:30:15.595 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:30:15.595 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:30:15.595 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:30:15.595 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:30:15.595 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:30:15.595 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:30:15.596 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:30:15.596 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:30:15.596 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:30:15.596 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:30:15.596 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:30:15.596 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:30:15.596 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:30:15.597 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:30:15.597 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:30:15.597 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:30:15.597 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:30:15.597 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:30:15.597 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:30:15.597 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:30:15.598 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:30:15.598 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:30:15.598 INFO:teuthology.orchestra.run.smithi130.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:30:15.598 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:15.598 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:30:15.598 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:30:15.598 INFO:teuthology.orchestra.run.smithi130.stdout:Install 21 Packages 2020-03-11T22:30:15.599 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 2 Packages 2020-03-11T22:30:15.599 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:15.599 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 51 M 2020-03-11T22:30:15.599 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:30:16.568 INFO:teuthology.orchestra.run.smithi130.stdout:(1/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 10 MB/s | 9.8 MB 00:00 2020-03-11T22:30:16.665 INFO:teuthology.orchestra.run.smithi130.stdout:(2/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 5.1 MB/s | 5.5 MB 00:01 2020-03-11T22:30:16.666 INFO:teuthology.orchestra.run.smithi130.stdout:(3/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 254 kB/s | 25 kB 00:00 2020-03-11T22:30:16.719 INFO:teuthology.orchestra.run.smithi130.stdout:(4/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 9.2 MB/s | 507 kB 00:00 2020-03-11T22:30:16.727 INFO:teuthology.orchestra.run.smithi130.stdout:(5/23): libradosstriper1-15.1.0-1941.g4acc112.e 6.0 MB/s | 377 kB 00:00 2020-03-11T22:30:16.778 INFO:teuthology.orchestra.run.smithi130.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 783 kB/s | 40 kB 00:00 2020-03-11T22:30:16.828 INFO:teuthology.orchestra.run.smithi130.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 966 kB/s | 49 kB 00:00 2020-03-11T22:30:16.887 INFO:teuthology.orchestra.run.smithi130.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 2.7 MB/s | 162 kB 00:00 2020-03-11T22:30:16.957 INFO:teuthology.orchestra.run.smithi130.stdout:(9/23): python3-rados-15.1.0-1941.g4acc112.el8. 4.9 MB/s | 352 kB 00:00 2020-03-11T22:30:17.029 INFO:teuthology.orchestra.run.smithi130.stdout:(10/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 4.2 MB/s | 309 kB 00:00 2020-03-11T22:30:17.096 INFO:teuthology.orchestra.run.smithi130.stdout:(11/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 1.6 MB/s | 113 kB 00:00 2020-03-11T22:30:17.130 INFO:teuthology.orchestra.run.smithi130.stdout:(12/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2020-03-11T22:30:17.134 INFO:teuthology.orchestra.run.smithi130.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 66 MB/s | 259 kB 00:00 2020-03-11T22:30:17.135 INFO:teuthology.orchestra.run.smithi130.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 29 MB/s | 44 kB 00:00 2020-03-11T22:30:17.137 INFO:teuthology.orchestra.run.smithi130.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 31 MB/s | 47 kB 00:00 2020-03-11T22:30:17.251 INFO:teuthology.orchestra.run.smithi130.stdout:(16/23): librgw2-15.1.0-1941.g4acc112.el8.x86_6 12 MB/s | 6.2 MB 00:00 2020-03-11T22:30:17.252 INFO:teuthology.orchestra.run.smithi130.stdout:(17/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.7 MB/s | 315 kB 00:00 2020-03-11T22:30:17.272 INFO:teuthology.orchestra.run.smithi130.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 3.8 MB/s | 75 kB 00:00 2020-03-11T22:30:17.276 INFO:teuthology.orchestra.run.smithi130.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 40 MB/s | 179 kB 00:00 2020-03-11T22:30:17.325 INFO:teuthology.orchestra.run.smithi130.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 790 kB/s | 59 kB 00:00 2020-03-11T22:30:17.724 INFO:teuthology.orchestra.run.smithi130.stdout:(21/23): ceph-common-15.1.0-1941.g4acc112.el8.x 9.7 MB/s | 21 MB 00:02 2020-03-11T22:30:17.809 INFO:teuthology.orchestra.run.smithi130.stdout:(22/23): librados2-15.1.0-1941.g4acc112.el8.x86 6.7 MB/s | 3.6 MB 00:00 2020-03-11T22:30:17.818 INFO:teuthology.orchestra.run.smithi130.stdout:(23/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 3.9 MB/s | 1.9 MB 00:00 2020-03-11T22:30:17.820 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:17.821 INFO:teuthology.orchestra.run.smithi130.stdout:Total 23 MB/s | 51 MB 00:02 2020-03-11T22:30:17.827 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:30:17.871 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:30:17.871 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:30:18.210 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:30:18.211 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:30:19.230 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:30:19.406 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:30:19.560 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:30:19.938 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:30:20.102 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:20.177 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:20.386 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:30:20.660 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:20.896 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:21.084 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:21.159 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:21.317 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:30:21.451 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:30:22.084 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:30:22.234 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:30:22.303 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:30:22.418 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:30:22.588 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:30:22.822 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:30:23.034 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:30:23.168 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:30:23.253 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:30:23.422 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:30:23.635 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:30:23.801 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:30:23.965 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:30:24.093 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:30:27.533 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:27.718 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:28.391 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:30:28.642 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:30:28.728 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:30:28.850 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:45.142 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:45.142 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2020-03-11T22:30:45.142 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2020-03-11T22:30:45.142 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2020-03-11T22:30:45.142 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /run 2020-03-11T22:30:45.142 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2020-03-11T22:30:45.143 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2020-03-11T22:30:45.143 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2020-03-11T22:30:45.143 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2020-03-11T22:30:45.143 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2020-03-11T22:30:45.143 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:46.439 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:30:46.503 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:30:46.503 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:30:46.658 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:30:46.658 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:47.347 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:47.347 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:30:47.348 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:30:47.348 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:30:47.348 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:30:47.348 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:30:47.349 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:30:47.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:30:47.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:30:47.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:30:47.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:30:47.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:30:47.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:30:47.351 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:30:47.351 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:30:47.352 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:30:47.352 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:30:47.352 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:30:47.352 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:30:47.352 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:30:47.352 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:30:47.352 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:30:47.353 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:30:47.353 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:30:47.353 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:30:48.018 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:30:48.018 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.019 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.020 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.021 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.021 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.021 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:48.021 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:30:48.021 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:30:48.021 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:30:48.021 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:30:48.022 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:30:48.022 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:30:48.022 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:30:48.022 INFO:teuthology.orchestra.run.smithi130.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:30:48.022 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:48.022 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:30:48.326 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-test 2020-03-11T22:30:48.734 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:34 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:30:49.100 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:30:49.101 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:30:49.102 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2020-03-11T22:30:49.102 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:30:49.102 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:30:49.102 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:30:49.102 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:30:49.102 INFO:teuthology.orchestra.run.smithi130.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:30:49.102 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:30:49.103 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:30:49.103 INFO:teuthology.orchestra.run.smithi130.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:30:49.103 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:30:49.103 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:49.103 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:30:49.103 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:30:49.103 INFO:teuthology.orchestra.run.smithi130.stdout:Install 6 Packages 2020-03-11T22:30:49.104 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:49.104 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 43 M 2020-03-11T22:30:49.104 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 184 M 2020-03-11T22:30:49.104 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:30:49.137 INFO:teuthology.orchestra.run.smithi130.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.7 MB/s | 161 kB 00:00 2020-03-11T22:30:49.141 INFO:teuthology.orchestra.run.smithi130.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.9 MB/s | 188 kB 00:00 2020-03-11T22:30:49.148 INFO:teuthology.orchestra.run.smithi130.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 27 MB/s | 298 kB 00:00 2020-03-11T22:30:49.150 INFO:teuthology.orchestra.run.smithi130.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 57 MB/s | 249 kB 00:00 2020-03-11T22:30:49.224 INFO:teuthology.orchestra.run.smithi130.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 920 kB/s | 69 kB 00:00 2020-03-11T22:30:51.356 INFO:teuthology.orchestra.run.smithi130.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 19 MB/s | 42 MB 00:02 2020-03-11T22:30:51.359 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:51.360 INFO:teuthology.orchestra.run.smithi130.stdout:Total 19 MB/s | 43 MB 00:02 2020-03-11T22:30:51.366 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:30:51.387 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:30:51.387 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:30:51.746 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:30:51.747 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:30:52.869 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:30:53.078 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:30:53.295 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:30:53.483 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:30:53.621 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:30:53.697 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:30:57.701 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:30:59.013 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:30:59.494 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:30:59.494 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:30:59.494 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:30:59.495 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:30:59.495 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:30:59.495 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:30:59.791 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:30:59.791 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:59.791 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:30:59.792 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:30:59.792 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:30:59.792 INFO:teuthology.orchestra.run.smithi130.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:30:59.792 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:30:59.792 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:30:59.924 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph 2020-03-11T22:31:00.319 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:46 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:00.700 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:00.705 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:00.705 INFO:teuthology.orchestra.run.smithi130.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:00.705 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:00.705 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:00.705 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:31:00.705 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:31:00.706 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:31:00.706 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:31:00.706 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:31:00.706 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:31:00.706 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:31:00.706 INFO:teuthology.orchestra.run.smithi130.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:31:00.706 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:31:00.707 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:31:00.707 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:31:00.707 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:31:00.707 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:31:00.707 INFO:teuthology.orchestra.run.smithi130.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:31:00.707 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:31:00.707 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:31:00.708 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:31:00.709 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:31:00.709 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:31:00.709 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:31:00.709 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:31:00.709 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:31:00.709 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:31:00.709 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:31:00.710 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:31:00.710 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:31:00.710 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:31:00.710 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:31:00.710 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2020-03-11T22:31:00.710 INFO:teuthology.orchestra.run.smithi130.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:31:00.710 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:31:00.711 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:00.711 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:00.711 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:00.713 INFO:teuthology.orchestra.run.smithi130.stdout:Install 36 Packages 2020-03-11T22:31:00.713 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:00.714 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 29 M 2020-03-11T22:31:00.714 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 105 M 2020-03-11T22:31:00.714 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:00.861 INFO:teuthology.orchestra.run.smithi130.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 43 kB/s | 6.4 kB 00:00 2020-03-11T22:31:01.157 INFO:teuthology.orchestra.run.smithi130.stdout:(2/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 4.5 MB/s | 2.0 MB 00:00 2020-03-11T22:31:01.411 INFO:teuthology.orchestra.run.smithi130.stdout:(3/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_6 7.9 MB/s | 4.4 MB 00:00 2020-03-11T22:31:01.473 INFO:teuthology.orchestra.run.smithi130.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.6 MB/s | 165 kB 00:00 2020-03-11T22:31:01.509 INFO:teuthology.orchestra.run.smithi130.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 5.8 MB/s | 214 kB 00:00 2020-03-11T22:31:01.528 INFO:teuthology.orchestra.run.smithi130.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 81 MB/s | 1.5 MB 00:00 2020-03-11T22:31:01.531 INFO:teuthology.orchestra.run.smithi130.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 58 MB/s | 157 kB 00:00 2020-03-11T22:31:01.533 INFO:teuthology.orchestra.run.smithi130.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 32 MB/s | 39 kB 00:00 2020-03-11T22:31:01.534 INFO:teuthology.orchestra.run.smithi130.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2020-03-11T22:31:01.535 INFO:teuthology.orchestra.run.smithi130.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 23 MB/s | 27 kB 00:00 2020-03-11T22:31:01.541 INFO:teuthology.orchestra.run.smithi130.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 76 MB/s | 457 kB 00:00 2020-03-11T22:31:01.544 INFO:teuthology.orchestra.run.smithi130.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 64 MB/s | 195 kB 00:00 2020-03-11T22:31:01.546 INFO:teuthology.orchestra.run.smithi130.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 28 MB/s | 34 kB 00:00 2020-03-11T22:31:01.548 INFO:teuthology.orchestra.run.smithi130.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 58 MB/s | 123 kB 00:00 2020-03-11T22:31:01.551 INFO:teuthology.orchestra.run.smithi130.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 64 MB/s | 176 kB 00:00 2020-03-11T22:31:01.609 INFO:teuthology.orchestra.run.smithi130.stdout:(16/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.5 MB/s | 145 kB 00:00 2020-03-11T22:31:01.631 INFO:teuthology.orchestra.run.smithi130.stdout:(17/36): python3-cherrypy-18.4.0-1.el8.noarch.r 17 MB/s | 385 kB 00:00 2020-03-11T22:31:01.639 INFO:teuthology.orchestra.run.smithi130.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.4 MB/s | 11 kB 00:00 2020-03-11T22:31:01.647 INFO:teuthology.orchestra.run.smithi130.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.2 MB/s | 18 kB 00:00 2020-03-11T22:31:01.658 INFO:teuthology.orchestra.run.smithi130.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 4.6 MB/s | 49 kB 00:00 2020-03-11T22:31:01.666 INFO:teuthology.orchestra.run.smithi130.stdout:(21/36): python3-more-itertools-7.2.0-3.el8.noa 6.7 MB/s | 59 kB 00:00 2020-03-11T22:31:01.678 INFO:teuthology.orchestra.run.smithi130.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 23 MB/s | 283 kB 00:00 2020-03-11T22:31:01.687 INFO:teuthology.orchestra.run.smithi130.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-03-11T22:31:01.697 INFO:teuthology.orchestra.run.smithi130.stdout:(24/36): python3-simplegeneric-0.8.1-17.el8.noa 1.8 MB/s | 19 kB 00:00 2020-03-11T22:31:01.706 INFO:teuthology.orchestra.run.smithi130.stdout:(25/36): python3-singledispatch-3.4.0.3-18.el8. 2.7 MB/s | 24 kB 00:00 2020-03-11T22:31:01.715 INFO:teuthology.orchestra.run.smithi130.stdout:(26/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.5 MB/s | 30 kB 00:00 2020-03-11T22:31:01.724 INFO:teuthology.orchestra.run.smithi130.stdout:(27/36): python3-trustme-0.5.2-4.el8.noarch.rpm 2.9 MB/s | 27 kB 00:00 2020-03-11T22:31:01.734 INFO:teuthology.orchestra.run.smithi130.stdout:(28/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.2 MB/s | 23 kB 00:00 2020-03-11T22:31:01.806 INFO:teuthology.orchestra.run.smithi130.stdout:(29/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_ 1.3 MB/s | 1.4 MB 00:01 2020-03-11T22:31:01.809 INFO:teuthology.orchestra.run.smithi130.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 583 kB/s | 44 kB 00:00 2020-03-11T22:31:01.847 INFO:teuthology.orchestra.run.smithi130.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 1.1 MB/s | 40 kB 00:00 2020-03-11T22:31:01.941 INFO:teuthology.orchestra.run.smithi130.stdout:(32/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 21 MB/s | 17 MB 00:00 2020-03-11T22:31:01.942 INFO:teuthology.orchestra.run.smithi130.stdout:(33/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2020-03-11T22:31:01.960 INFO:teuthology.orchestra.run.smithi130.stdout:(34/36): python3-waitress-1.2.1-2.el8.1.noarch. 2.1 MB/s | 241 kB 00:00 2020-03-11T22:31:01.962 INFO:teuthology.orchestra.run.smithi130.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.3 MB/s | 85 kB 00:00 2020-03-11T22:31:02.054 INFO:teuthology.orchestra.run.smithi130.stdout:(36/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.2 MB/s | 251 kB 00:00 2020-03-11T22:31:02.057 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:02.058 INFO:teuthology.orchestra.run.smithi130.stdout:Total 22 MB/s | 29 MB 00:01 2020-03-11T22:31:02.077 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:02.115 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:02.115 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:02.476 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:02.480 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:03.732 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:03.956 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:31:04.413 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:31:04.562 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:31:04.754 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:31:04.936 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:31:05.110 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:31:05.266 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:31:05.439 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:31:05.598 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:31:05.725 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:31:05.908 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:31:06.093 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:31:06.315 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:31:06.546 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:31:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:31:06.988 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:31:07.197 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:31:07.438 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:31:07.632 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:31:08.008 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:31:08.253 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:31:08.441 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:31:08.613 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:31:08.774 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:31:09.034 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:31:09.238 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:31:09.481 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:31:09.795 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:31:10.067 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:31:10.402 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:31:10.544 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:31:11.963 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:31:12.102 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:31:12.491 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:31:12.619 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:31:12.811 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:31:12.968 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:31:12.987 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:31:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:31:13.404 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:31:13.404 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:31:13.405 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:31:13.405 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:31:13.405 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:31:13.405 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:31:13.405 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:31:13.405 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:31:13.406 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:31:13.406 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:31:13.406 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:31:13.406 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:31:13.407 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:31:13.407 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:31:13.407 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:31:13.407 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:31:13.407 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:31:13.408 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:31:13.408 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:31:13.408 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:31:13.409 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:31:13.409 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:31:13.409 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:31:13.409 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:31:13.409 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:31:13.409 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:31:13.411 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:31:13.411 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:31:13.411 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:31:13.411 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:31:13.411 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:31:13.412 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:31:13.412 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:31:13.412 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:31:13.412 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:31:13.412 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:31:14.123 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:31:14.123 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:14.123 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:14.123 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:14.124 INFO:teuthology.orchestra.run.smithi130.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:31:14.125 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:31:14.125 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:31:14.125 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:31:14.125 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:31:14.125 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:31:14.125 INFO:teuthology.orchestra.run.smithi130.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:31:14.125 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:31:14.126 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:31:14.126 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:31:14.126 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:31:14.126 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:31:14.126 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:31:14.126 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:31:14.126 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:31:14.127 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:31:14.128 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:31:14.128 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:31:14.128 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:31:14.128 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:31:14.128 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:31:14.128 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:14.129 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:14.432 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-base 2020-03-11T22:31:14.836 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:15.084 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:15.191 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:15.193 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:31:15.193 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:15.331 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install cephadm 2020-03-11T22:31:15.724 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:16.108 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:16.109 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:16.109 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2020-03-11T22:31:16.109 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:16.109 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:16.109 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:31:16.109 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:16.109 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:16.110 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:16.110 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2020-03-11T22:31:16.110 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:16.110 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 36 k 2020-03-11T22:31:16.111 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 129 k 2020-03-11T22:31:16.111 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:16.297 INFO:teuthology.orchestra.run.smithi130.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 190 kB/s | 36 kB 00:00 2020-03-11T22:31:16.298 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:16.298 INFO:teuthology.orchestra.run.smithi130.stdout:Total 189 kB/s | 36 kB 00:00 2020-03-11T22:31:16.300 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:16.307 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:16.307 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:16.309 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:16.309 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:16.369 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:17.633 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:17.881 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:18.109 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:18.109 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:18.110 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:18.110 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:18.110 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:18.110 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:18.241 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:31:18.642 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:18.994 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:18.995 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:18.995 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2020-03-11T22:31:18.995 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:18.995 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:18.995 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:31:18.996 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:18.996 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:18.996 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:18.996 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2020-03-11T22:31:18.996 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:18.997 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 203 k 2020-03-11T22:31:18.997 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 743 k 2020-03-11T22:31:18.997 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:19.290 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 690 kB/s | 203 kB 00:00 2020-03-11T22:31:19.291 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:19.291 INFO:teuthology.orchestra.run.smithi130.stdout:Total 688 kB/s | 203 kB 00:00 2020-03-11T22:31:19.293 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:19.305 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:19.306 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:19.351 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:19.351 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:19.459 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:19.584 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:19.896 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:20.281 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:31:20.282 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:20.282 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:20.282 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:20.282 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:20.282 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:20.538 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr 2020-03-11T22:31:20.941 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:06 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:21.224 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:21.331 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:21.332 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:31:21.332 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:21.456 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:31:21.848 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:22.237 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:22.238 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:22.238 INFO:teuthology.orchestra.run.smithi130.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:22.238 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:22.238 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:22.238 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:31:22.238 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:31:22.239 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:31:22.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:31:22.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:31:22.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:31:22.239 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:22.239 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:22.239 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:22.240 INFO:teuthology.orchestra.run.smithi130.stdout:Install 5 Packages 2020-03-11T22:31:22.240 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:22.240 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 4.2 M 2020-03-11T22:31:22.240 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 19 M 2020-03-11T22:31:22.241 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:22.277 INFO:teuthology.orchestra.run.smithi130.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2020-03-11T22:31:22.352 INFO:teuthology.orchestra.run.smithi130.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 480 kB/s | 34 kB 00:00 2020-03-11T22:31:22.379 INFO:teuthology.orchestra.run.smithi130.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 7.3 MB/s | 197 kB 00:00 2020-03-11T22:31:22.403 INFO:teuthology.orchestra.run.smithi130.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 137 kB/s | 22 kB 00:00 2020-03-11T22:31:22.721 INFO:teuthology.orchestra.run.smithi130.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 8.0 MB/s | 3.9 MB 00:00 2020-03-11T22:31:22.723 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:22.724 INFO:teuthology.orchestra.run.smithi130.stdout:Total 8.6 MB/s | 4.2 MB 00:00 2020-03-11T22:31:22.729 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:22.737 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:22.737 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:22.891 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:22.908 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:23.105 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:23.340 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:31:23.526 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:31:23.707 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:31:24.712 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:31:24.983 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:31:25.186 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:31:25.187 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:31:25.187 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:31:25.187 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:31:25.187 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:31:25.479 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:31:25.480 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:25.480 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:25.480 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:25.480 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:25.480 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:31:25.480 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:31:25.481 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:31:25.481 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:25.481 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:27.284 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:31:27.689 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:13 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:28.053 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:28.055 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:28.055 INFO:teuthology.orchestra.run.smithi130.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:28.055 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:28.055 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:28.055 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:31:28.056 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:31:28.056 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:31:28.056 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:31:28.056 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:31:28.056 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:31:28.057 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:31:28.057 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:31:28.057 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:31:28.057 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:28.058 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:28.058 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:28.058 INFO:teuthology.orchestra.run.smithi130.stdout:Install 8 Packages 2020-03-11T22:31:28.058 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:28.059 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 37 M 2020-03-11T22:31:28.059 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 205 M 2020-03-11T22:31:28.060 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:28.172 INFO:teuthology.orchestra.run.smithi130.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 38 MB/s | 4.3 MB 00:00 2020-03-11T22:31:28.198 INFO:teuthology.orchestra.run.smithi130.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2020-03-11T22:31:28.206 INFO:teuthology.orchestra.run.smithi130.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 31 MB/s | 225 kB 00:00 2020-03-11T22:31:28.248 INFO:teuthology.orchestra.run.smithi130.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 49 MB/s | 3.7 MB 00:00 2020-03-11T22:31:28.263 INFO:teuthology.orchestra.run.smithi130.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 44 MB/s | 638 kB 00:00 2020-03-11T22:31:28.268 INFO:teuthology.orchestra.run.smithi130.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 32 MB/s | 169 kB 00:00 2020-03-11T22:31:28.471 INFO:teuthology.orchestra.run.smithi130.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 55 MB/s | 15 MB 00:00 2020-03-11T22:31:30.444 INFO:teuthology.orchestra.run.smithi130.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 3.6 MB/s | 8.6 MB 00:02 2020-03-11T22:31:30.447 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:30.447 INFO:teuthology.orchestra.run.smithi130.stdout:Total 15 MB/s | 37 MB 00:02 2020-03-11T22:31:30.458 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:30.478 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:30.478 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:30.771 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:30.771 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:31.318 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:31.499 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:31:31.636 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:31:31.782 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:31:32.317 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:31:32.449 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:32.974 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:33.123 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:31:33.829 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:31:34.135 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:31:34.365 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:31:36.312 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:31:37.565 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:31:37.956 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:31:38.189 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:31:38.190 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:31:38.190 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:31:38.190 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:31:38.190 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:31:38.190 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:31:38.190 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:31:38.190 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:31:38.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:31:38.448 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:38.448 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:38.448 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:38.449 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:31:38.449 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:31:38.449 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:31:38.449 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:31:38.449 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:31:38.450 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:31:38.450 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:31:38.450 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:38.450 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:38.578 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:31:38.974 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:39.339 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:39.341 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:39.341 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2020-03-11T22:31:39.341 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:39.341 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:39.341 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-noarch 24 k 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:31:39.342 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:31:39.343 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:31:39.343 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:31:39.343 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:31:39.343 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:31:39.343 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:31:39.343 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:31:39.343 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:39.344 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:39.344 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:39.344 INFO:teuthology.orchestra.run.smithi130.stdout:Install 13 Packages 2020-03-11T22:31:39.344 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:39.345 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 1.9 M 2020-03-11T22:31:39.345 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 23 M 2020-03-11T22:31:39.345 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:39.374 INFO:teuthology.orchestra.run.smithi130.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 881 kB/s | 27 kB 00:00 2020-03-11T22:31:39.378 INFO:teuthology.orchestra.run.smithi130.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 37 MB/s | 126 kB 00:00 2020-03-11T22:31:39.378 INFO:teuthology.orchestra.run.smithi130.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 581 kB/s | 20 kB 00:00 2020-03-11T22:31:39.380 INFO:teuthology.orchestra.run.smithi130.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 42 MB/s | 110 kB 00:00 2020-03-11T22:31:39.387 INFO:teuthology.orchestra.run.smithi130.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 28 MB/s | 43 kB 00:00 2020-03-11T22:31:39.387 INFO:teuthology.orchestra.run.smithi130.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 17 MB/s | 155 kB 00:00 2020-03-11T22:31:39.434 INFO:teuthology.orchestra.run.smithi130.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 698 kB/s | 33 kB 00:00 2020-03-11T22:31:39.444 INFO:teuthology.orchestra.run.smithi130.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 291 kB/s | 16 kB 00:00 2020-03-11T22:31:39.467 INFO:teuthology.orchestra.run.smithi130.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.4 MB/s | 81 kB 00:00 2020-03-11T22:31:39.483 INFO:teuthology.orchestra.run.smithi130.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.5 MB/s | 73 kB 00:00 2020-03-11T22:31:39.499 INFO:teuthology.orchestra.run.smithi130.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.8 MB/s | 61 kB 00:00 2020-03-11T22:31:39.532 INFO:teuthology.orchestra.run.smithi130.stdout:(12/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 129 kB/s | 24 kB 00:00 2020-03-11T22:31:39.579 INFO:teuthology.orchestra.run.smithi130.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 8.2 MB/s | 1.1 MB 00:00 2020-03-11T22:31:39.581 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:39.581 INFO:teuthology.orchestra.run.smithi130.stdout:Total 7.8 MB/s | 1.9 MB 00:00 2020-03-11T22:31:39.587 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:39.601 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:39.601 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:39.692 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:39.693 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:40.159 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:40.400 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:31:40.611 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:31:40.800 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:31:40.987 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:31:41.153 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:31:41.367 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:31:41.634 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:31:41.843 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:31:42.851 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:31:43.086 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:31:43.287 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:31:44.231 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:31:44.405 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:31:44.626 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:31:44.626 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:31:44.627 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:31:44.627 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:31:44.627 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:31:44.627 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:31:44.627 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:31:44.627 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:31:44.627 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:31:44.628 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:31:44.628 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:31:44.628 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:31:44.628 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:31:44.938 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:31:44.939 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:44.939 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:44.939 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:44.939 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:31:44.940 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:31:44.940 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:31:44.940 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:31:44.940 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:31:44.940 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:31:44.941 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:31:44.941 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:31:44.941 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:31:44.941 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:31:44.942 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:31:44.942 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:31:44.942 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:44.942 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:45.100 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:31:45.498 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:31 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:45.867 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:45.868 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:45.868 INFO:teuthology.orchestra.run.smithi130.stdout: Package ArchVersion Repository Size 2020-03-11T22:31:45.868 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:45.868 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:45.868 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:31:45.869 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:31:45.869 INFO:teuthology.orchestra.run.smithi130.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:31:45.869 INFO:teuthology.orchestra.run.smithi130.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:31:45.869 INFO:teuthology.orchestra.run.smithi130.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:31:45.869 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:45.869 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:45.869 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:45.870 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2020-03-11T22:31:45.870 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:45.870 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 461 k 2020-03-11T22:31:45.870 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 1.4 M 2020-03-11T22:31:45.871 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:45.926 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 940 kB/s | 48 kB 00:00 2020-03-11T22:31:45.946 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.3 MB/s | 356 kB 00:00 2020-03-11T22:31:45.974 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 449 kB/s | 21 kB 00:00 2020-03-11T22:31:46.067 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 194 kB/s | 36 kB 00:00 2020-03-11T22:31:46.067 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:46.068 INFO:teuthology.orchestra.run.smithi130.stdout:Total 2.4 MB/s | 461 kB 00:00 2020-03-11T22:31:46.071 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:46.077 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:46.078 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:46.103 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:46.104 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:46.262 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:46.497 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:31:46.777 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:31:46.977 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:31:47.121 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:31:47.319 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:31:47.319 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:31:47.319 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:31:47.319 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:31:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:31:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:47.554 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:31:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:31:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:31:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:31:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:47.555 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:47.701 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-fuse 2020-03-11T22:31:48.098 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:48.450 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:48.451 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:48.451 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2020-03-11T22:31:48.451 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:48.451 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:48.452 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:31:48.452 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2020-03-11T22:31:48.452 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:31:48.452 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:31:48.452 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:48.452 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:48.453 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:48.453 INFO:teuthology.orchestra.run.smithi130.stdout:Install 3 Packages 2020-03-11T22:31:48.453 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:48.454 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 676 k 2020-03-11T22:31:48.454 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 2.1 M 2020-03-11T22:31:48.454 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:48.484 INFO:teuthology.orchestra.run.smithi130.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2020-03-11T22:31:48.487 INFO:teuthology.orchestra.run.smithi130.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 623 kB/s | 21 kB 00:00 2020-03-11T22:31:48.784 INFO:teuthology.orchestra.run.smithi130.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.7 MB/s | 572 kB 00:00 2020-03-11T22:31:48.785 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:48.786 INFO:teuthology.orchestra.run.smithi130.stdout:Total 2.0 MB/s | 676 kB 00:00 2020-03-11T22:31:48.789 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:48.810 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:48.810 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:48.873 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:48.874 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:49.038 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:49.039 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:31:49.039 INFO:teuthology.orchestra.run.smithi130.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:31:49.039 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:49.420 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:31:49.903 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:31:50.206 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:31:50.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:31:50.206 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:31:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:31:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:50.491 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:31:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:31:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:50.492 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:50.776 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install librados-devel 2020-03-11T22:31:51.181 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:51.577 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:51.577 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:51.577 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2020-03-11T22:31:51.578 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:51.578 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:51.578 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:31:51.578 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:51.578 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:51.578 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:51.578 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2020-03-11T22:31:51.579 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:51.579 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 141 k 2020-03-11T22:31:51.579 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 487 k 2020-03-11T22:31:51.579 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 287 kB/s | 141 kB 00:00 2020-03-11T22:31:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:52.070 INFO:teuthology.orchestra.run.smithi130.stdout:Total 286 kB/s | 141 kB 00:00 2020-03-11T22:31:52.072 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:52.079 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:52.079 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:52.129 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:52.129 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:52.235 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:52.468 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:52.675 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:52.954 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:52.955 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:52.955 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:52.955 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:52.955 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:52.955 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:53.250 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs2 2020-03-11T22:31:53.655 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:53.920 INFO:teuthology.orchestra.run.smithi130.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:54.026 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:54.027 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:31:54.027 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:54.183 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs-devel 2020-03-11T22:31:54.580 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:54.953 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:54.954 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:54.954 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2020-03-11T22:31:54.954 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:54.954 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:31:54.954 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:31:54.954 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:54.954 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:31:54.955 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:31:54.955 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2020-03-11T22:31:54.955 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:54.955 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 21 k 2020-03-11T22:31:54.955 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 76 k 2020-03-11T22:31:54.955 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:31:55.156 INFO:teuthology.orchestra.run.smithi130.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 106 kB/s | 21 kB 00:00 2020-03-11T22:31:55.156 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:55.156 INFO:teuthology.orchestra.run.smithi130.stdout:Total 105 kB/s | 21 kB 00:00 2020-03-11T22:31:55.158 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:31:55.162 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:31:55.162 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:31:55.164 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:31:55.165 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:31:55.394 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:31:55.551 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:55.737 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:55.944 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:55.944 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:55.945 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:31:55.945 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:55.945 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:31:55.945 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:56.107 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install librados2 2020-03-11T22:31:56.505 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:56.759 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:56.865 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:56.866 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:31:56.867 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:57.047 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install librbd1 2020-03-11T22:31:57.445 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:57.709 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:57.816 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:57.817 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:31:57.817 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:57.973 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rados 2020-03-11T22:31:58.378 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:58.660 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:58.766 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:58.767 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:31:58.767 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:58.916 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rgw 2020-03-11T22:31:59.315 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:31:59.576 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:31:59.683 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:31:59.684 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:31:59.684 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:31:59.867 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-cephfs 2020-03-11T22:32:00.267 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:32:00.525 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:32:00.632 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:32:00.633 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:32:00.633 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:32:00.926 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rbd 2020-03-11T22:32:01.327 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:32:01.583 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:32:01.688 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:32:01.689 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2020-03-11T22:32:01.689 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:32:01.829 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-fuse 2020-03-11T22:32:02.230 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:32:02.590 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:32:02.590 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:02.590 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2020-03-11T22:32:02.591 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:02.591 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:32:02.591 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:32:02.591 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:02.591 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:32:02.591 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:02.592 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2020-03-11T22:32:02.592 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:02.592 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 72 k 2020-03-11T22:32:02.592 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 230 k 2020-03-11T22:32:02.593 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:32:03.019 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 169 kB/s | 72 kB 00:00 2020-03-11T22:32:03.020 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:03.020 INFO:teuthology.orchestra.run.smithi130.stdout:Total 168 kB/s | 72 kB 00:00 2020-03-11T22:32:03.022 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:32:03.033 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:32:03.033 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:32:03.082 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:32:03.082 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:32:03.192 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:32:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:03.562 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:03.802 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:03.802 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:03.802 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:32:03.802 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:03.803 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:03.803 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:32:04.108 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-mirror 2020-03-11T22:32:04.511 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:32:04.882 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:32:04.883 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:04.883 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2020-03-11T22:32:04.883 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:04.883 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:32:04.884 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:32:04.884 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:04.884 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:32:04.884 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:04.884 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2020-03-11T22:32:04.884 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:04.885 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 2.7 M 2020-03-11T22:32:04.885 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 12 M 2020-03-11T22:32:04.885 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:32:05.267 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 7.0 MB/s | 2.7 MB 00:00 2020-03-11T22:32:05.268 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:05.269 INFO:teuthology.orchestra.run.smithi130.stdout:Total 7.0 MB/s | 2.7 MB 00:00 2020-03-11T22:32:05.270 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:32:05.283 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:32:05.283 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:32:05.341 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:32:05.341 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:32:05.696 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:32:05.868 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:06.179 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:06.750 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:32:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:06.750 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:32:08.385 INFO:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-nbd 2020-03-11T22:32:08.798 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:54 ago on Wed 11 Mar 2020 10:30:14 PM UTC. 2020-03-11T22:32:09.207 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2020-03-11T22:32:09.208 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:09.209 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2020-03-11T22:32:09.209 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:09.209 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2020-03-11T22:32:09.209 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:32:09.209 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:09.209 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2020-03-11T22:32:09.210 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2020-03-11T22:32:09.210 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2020-03-11T22:32:09.210 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:09.210 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 141 k 2020-03-11T22:32:09.210 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 526 k 2020-03-11T22:32:09.210 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2020-03-11T22:32:09.830 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 227 kB/s | 141 kB 00:00 2020-03-11T22:32:09.830 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:09.831 INFO:teuthology.orchestra.run.smithi130.stdout:Total 227 kB/s | 141 kB 00:00 2020-03-11T22:32:09.833 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2020-03-11T22:32:09.845 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2020-03-11T22:32:09.845 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2020-03-11T22:32:09.895 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2020-03-11T22:32:09.895 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2020-03-11T22:32:10.026 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2020-03-11T22:32:10.199 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:10.390 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:10.653 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:10.653 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:10.653 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2020-03-11T22:32:10.653 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:10.653 INFO:teuthology.orchestra.run.smithi130.stdout: 2020-03-11T22:32:10.653 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2020-03-11T22:32:10.983 DEBUG:teuthology.parallel:result is None 2020-03-11T22:32:10.984 INFO:teuthology.orchestra.run.smithi130:> sudo lsb_release -is 2020-03-11T22:32:11.020 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS 2020-03-11T22:32:11.021 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:32:11.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:32:11.237 INFO:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:32:11.256 INFO:teuthology.orchestra.run.smithi130.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:32:11.257 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:32:11.257 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:32:11.257 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:32:11.263 INFO:teuthology.orchestra.run.smithi130:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:32:11.324 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:32:11.328 INFO:teuthology.orchestra.run.smithi130:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:32:11.394 INFO:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:32:11.460 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:32:11.460 INFO:teuthology.orchestra.run.smithi130:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:32:11.547 INFO:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:32:11.613 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:32:11.632 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:32:11.632 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:32:11.632 INFO:tasks.workunit:timeout=3h 2020-03-11T22:32:11.632 INFO:tasks.workunit:cleanup=True 2020-03-11T22:32:11.633 INFO:teuthology.orchestra.run.smithi130:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:32:11.666 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:32:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:32:11.667 INFO:teuthology.orchestra.run.smithi130:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:32:11.720 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:32:11.720 INFO:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:32:11.775 INFO:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:32:11.881 INFO:tasks.workunit.client.0.smithi130.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:33:49.001 INFO:tasks.workunit.client.0.smithi130.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:33:49.001 INFO:tasks.workunit.client.0.smithi130.stderr: 2020-03-11T22:33:49.001 INFO:tasks.workunit.client.0.smithi130.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:33:49.001 INFO:tasks.workunit.client.0.smithi130.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:33:49.002 INFO:tasks.workunit.client.0.smithi130.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:33:49.002 INFO:tasks.workunit.client.0.smithi130.stderr: 2020-03-11T22:33:49.002 INFO:tasks.workunit.client.0.smithi130.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:33:49.002 INFO:tasks.workunit.client.0.smithi130.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:33:49.002 INFO:tasks.workunit.client.0.smithi130.stderr: 2020-03-11T22:33:49.002 INFO:tasks.workunit.client.0.smithi130.stderr: git checkout -b 2020-03-11T22:33:49.002 INFO:tasks.workunit.client.0.smithi130.stderr: 2020-03-11T22:33:49.003 INFO:tasks.workunit.client.0.smithi130.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:33:49.006 INFO:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:33:49.077 DEBUG:teuthology.orchestra.remote:smithi130:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:33:49.126 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:33:49.126 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:33:49.126 INFO:teuthology.orchestra.run.smithi130:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:33:49.177 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:33:49.186 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:33:49.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:33:49.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:33:49.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:33:49.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:33:49.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:33:49.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:33:49.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:33:49.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:33:49.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:33:49.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:33:49.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:33:49.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:33:49.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:33:49.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:33:49.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:33:49.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:33:49.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:33:49.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:33:49.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:33:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:33:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:33:49.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:33:49.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:33:49.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:33:49.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:33:49.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:33:49.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:33:49.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:33:49.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:33:49.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:33:49.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:33:49.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:33:49.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:33:49.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:33:49.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:33:49.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none ' 2020-03-11T22:33:49.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:33:49.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:33:49.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:33:49.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:49.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:33:49.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:33:49.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:33:49.202 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_divergent_2 2020-03-11T22:33:49.202 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_divergent_3 2020-03-11T22:33:49.202 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_divergent_ec' 2020-03-11T22:33:49.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:33:49.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:33:49.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:33:49.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:33:49.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:33:49.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:33:49.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:33:49.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:33:49.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:33:49.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:33:49.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:33:49.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:33:49.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:33:49.205 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:33:49.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:33:49.205 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:33:49.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:33:49.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:33:49.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:33:49.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:33:49.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:33:49.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:33:49.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:33:49.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:33:49.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:33:49.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:33:49.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:33:49.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:49.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:49.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:33:49.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:33:49.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:33:49.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:33:49.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:33:49.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:49.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:49.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17424 2020-03-11T22:33:49.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:33:49.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:33:49.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:33:49.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:33:49.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:33:49.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:33:49.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:33:49.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:33:49.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:33:49.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:33:49.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:33:49.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:33:49.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:33:49.222 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:33:49.222 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:33:49.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:33:49.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:33:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:33:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:33:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:33:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:33:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:33:49.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:33:49.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:49.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:49.595 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:49.595 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:49.595 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:49.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:33:49.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:33:49.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:33:49.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:33:49.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:33:49.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:33:49.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:33:49.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:33:49.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:33:49.766 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:33:49.766 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:33:49.766 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:49.766 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:49.766 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:49.766 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:33:49.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:33:49.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get fsid 2020-03-11T22:33:49.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:33:49.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:33:49.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:33:49.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:33:49.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:33:49.851 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:33:49.851 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:33:49.852 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:33:49.852 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:49.852 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:49.852 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:49.852 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:33:49.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:33:49.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get mon_host 2020-03-11T22:33:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:33:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:33:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:33:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:33:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:33:49.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:33:49.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:33:50.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:33:50.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:50.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:50.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:50.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:50.804 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:50.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:33:50.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:33:50.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:33:50.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:50.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:33:50.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:50.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:50.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:33:50.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:50.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:33:50.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:50.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:50.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:50.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:50.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:33:50.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:33:50.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:50.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:50.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:33:50.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:33:50.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:50.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:50.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:50.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:50.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:50.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:50.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:50.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:50.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:50.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:50.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:33:50.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:50.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f460e2f8-4c70-4627-a3ef-aff3188231e6 2020-03-11T22:33:50.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 f460e2f8-4c70-4627-a3ef-aff3188231e6' 2020-03-11T22:33:50.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:50.838 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 f460e2f8-4c70-4627-a3ef-aff3188231e6 2020-03-11T22:33:50.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBOZ2lesGeAMhAA1lBN35bz5aVBNQjpSlMZCA== 2020-03-11T22:33:50.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBOZ2lesGeAMhAA1lBN35bz5aVBNQjpSlMZCA=="}' 2020-03-11T22:33:50.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f460e2f8-4c70-4627-a3ef-aff3188231e6 -i td/divergent-priors/0/new.json 2020-03-11T22:33:51.185 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:33:51.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:33:51.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBOZ2lesGeAMhAA1lBN35bz5aVBNQjpSlMZCA== --osd-uuid f460e2f8-4c70-4627-a3ef-aff3188231e6 2020-03-11T22:33:51.224 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:51.221+0000 7f2ea2bc1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:51.226 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:51.223+0000 7f2ea2bc1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:51.227 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:51.224+0000 7f2ea2bc1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:51.228 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:51.225+0000 7f2ea2bc1ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:51.228 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:51.225+0000 7f2ea2bc1ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:51.228 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:51.225+0000 7f2ea2bc1ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:33:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:51.258+0000 7f2ea2bc1ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:33:54.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:33:54.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:33:54.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:33:54.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:33:54.249 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:33:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:33:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:33:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:33:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:33:54.507 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:33:54.532 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:54.529+0000 7f41e287aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:54.546 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:54.543+0000 7f41e287aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:54.548 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:54.545+0000 7f41e287aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:54.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:33:54.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:54.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:33:54.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:54.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:54.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:54.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:54.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:54.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:54.751 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:33:55.064 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:55.060+0000 7f41e287aec0 -1 Falling back to public interface 2020-03-11T22:33:55.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:55.803 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:55.800+0000 7f41e287aec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:33:56.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:56.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:56.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:56.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:56.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:56.098 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:33:56.486 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17806,v1:127.0.0.1:6803/17806] [v2:127.0.0.1:6804/17806,v1:127.0.0.1:6805/17806] exists,up f460e2f8-4c70-4627-a3ef-aff3188231e6 2020-03-11T22:33:56.486 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 bec02fee-4bcc-4b9d-a58d-cfa0750b447c 2020-03-11T22:33:56.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:33:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:33:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:33:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:33:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:33:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:33:56.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:56.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:33:56.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:56.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:33:56.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:56.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:56.489 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:56.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:56.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:33:56.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:33:56.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:33:56.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:56.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:33:56.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:33:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:33:56.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:33:56.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bec02fee-4bcc-4b9d-a58d-cfa0750b447c 2020-03-11T22:33:56.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 bec02fee-4bcc-4b9d-a58d-cfa0750b447c' 2020-03-11T22:33:56.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:33:56.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBUZ2le86zgHRAAZp6WmTLaoM5RY8isvCZQ8g== 2020-03-11T22:33:56.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBUZ2le86zgHRAAZp6WmTLaoM5RY8isvCZQ8g=="}' 2020-03-11T22:33:56.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bec02fee-4bcc-4b9d-a58d-cfa0750b447c -i td/divergent-priors/1/new.json 2020-03-11T22:33:57.019 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:33:57.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:33:57.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBUZ2le86zgHRAAZp6WmTLaoM5RY8isvCZQ8g== --osd-uuid bec02fee-4bcc-4b9d-a58d-cfa0750b447c 2020-03-11T22:33:57.057 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:57.054+0000 7f3f28d50ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:57.060 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:57.057+0000 7f3f28d50ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:57.061 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:57.057+0000 7f3f28d50ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:57.061 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:57.058+0000 7f3f28d50ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:57.061 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:57.058+0000 7f3f28d50ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:57.062 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:57.058+0000 7f3f28d50ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:33:57.127 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:33:57.124+0000 7f3f28d50ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:34:00.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:34:00.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:34:00.128 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:34:00.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:34:00.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:34:00.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:34:00.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:00.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:34:00.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:34:00.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:34:00.584 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:34:00.609 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:00.606+0000 7f29beae2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:00.612 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:00.608+0000 7f29beae2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:00.612 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:00.609+0000 7f29beae2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:00.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:34:00.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:00.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:34:00.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:00.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:00.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:00.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:00.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:00.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:34:00.930 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:34:01.129 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:01.125+0000 7f29beae2ec0 -1 Falling back to public interface 2020-03-11T22:34:01.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:01.900 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:01.897+0000 7f29beae2ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:34:02.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:02.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:02.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:02.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:02.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:34:02.288 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:34:02.648 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18216,v1:127.0.0.1:6811/18216] [v2:127.0.0.1:6812/18216,v1:127.0.0.1:6813/18216] exists,up bec02fee-4bcc-4b9d-a58d-cfa0750b447c 2020-03-11T22:34:02.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:02.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:02.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:02.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:34:02.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:34:02.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:34:02.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:34:02.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:34:02.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:34:02.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:34:02.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:02.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:02.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:02.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:02.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:34:02.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:34:02.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:02.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:34:02.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:02.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:34:02.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:02.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:02.652 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:02.652 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:02.652 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:34:02.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:34:02.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:02.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:02.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:34:02.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:34:02.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:34:02.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:34:02.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8a6256f9-8a8b-47fd-8dc9-81d6ed0db004 2020-03-11T22:34:02.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 8a6256f9-8a8b-47fd-8dc9-81d6ed0db004' 2020-03-11T22:34:02.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:34:02.656 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 8a6256f9-8a8b-47fd-8dc9-81d6ed0db004 2020-03-11T22:34:02.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBaZ2lep4ywJxAAI0UAF9rA6i7Ls6UKYAuUEg== 2020-03-11T22:34:02.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBaZ2lep4ywJxAAI0UAF9rA6i7Ls6UKYAuUEg=="}' 2020-03-11T22:34:02.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8a6256f9-8a8b-47fd-8dc9-81d6ed0db004 -i td/divergent-priors/2/new.json 2020-03-11T22:34:03.136 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:34:03.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:34:03.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBaZ2lep4ywJxAAI0UAF9rA6i7Ls6UKYAuUEg== --osd-uuid 8a6256f9-8a8b-47fd-8dc9-81d6ed0db004 2020-03-11T22:34:03.175 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:03.172+0000 7fce41564ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:03.177 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:03.174+0000 7fce41564ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:03.179 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:03.174+0000 7fce41564ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:03.179 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:03.175+0000 7fce41564ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:34:03.179 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:03.175+0000 7fce41564ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:34:03.179 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:03.175+0000 7fce41564ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:34:03.256 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:03.253+0000 7fce41564ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:34:06.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:34:06.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:34:06.257 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:34:06.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:34:06.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:34:06.732 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:34:06.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:34:06.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:06.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:34:06.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:34:06.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:34:06.758 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:06.755+0000 7f505b213ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:06.760 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:06.757+0000 7f505b213ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:06.762 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:06.759+0000 7f505b213ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:07.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:34:07.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:07.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:34:07.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:07.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:07.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:07.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:07.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:07.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:07.080 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:34:07.278 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:07.274+0000 7f505b213ec0 -1 Falling back to public interface 2020-03-11T22:34:07.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:08.020 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:08.017+0000 7f505b213ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:34:08.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:08.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:08.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:08.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:08.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:08.423 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:34:08.775 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/18571,v1:127.0.0.1:6819/18571] [v2:127.0.0.1:6820/18571,v1:127.0.0.1:6821/18571] exists,up 8a6256f9-8a8b-47fd-8dc9-81d6ed0db004 2020-03-11T22:34:08.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:08.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:08.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:08.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:34:09.480 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2020-03-11T22:34:09.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:34:10.173 INFO:tasks.workunit.client.0.smithi130.stderr:noin is set 2020-03-11T22:34:10.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:34:10.799 INFO:tasks.workunit.client.0.smithi130.stderr:nodown is set 2020-03-11T22:34:10.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:34:10.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:34:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2020-03-11T22:34:11.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:34:12.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:34:12.968 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 3 2020-03-11T22:34:12.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:34:13.645 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 min_size to 2 2020-03-11T22:34:13.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:34:13.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:13.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:13.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:13.999 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:34:13.999 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:34:14.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:14.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:14.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:14.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:34:14.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:34:14.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:14.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:14.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:34:14.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:34:14.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:14.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:14.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-03-11T22:34:14.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-60129542148' 2020-03-11T22:34:14.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:14.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:34:14.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:14.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:14.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:34:14.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:14.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:34:14.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:34:14.391 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:34:14.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:14.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:34:14.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:14.751 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:34:14.751 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:14.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:14.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:34:14.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:14.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:34:14.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:34:14.754 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:34:14.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:15.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:34:15.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:15.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542148 2020-03-11T22:34:15.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:15.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:15.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542148 2020-03-11T22:34:15.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:15.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542148 2020-03-11T22:34:15.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-03-11T22:34:15.120 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 60129542148 2020-03-11T22:34:15.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:15.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542148 2020-03-11T22:34:15.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:34:15.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:15.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:15.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:15.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:15.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:34:15.464 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:15.464 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:15.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:15.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:15.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:15.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:15.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:15.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:15.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:15.554 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:15.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:15.896 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:34:15.896 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:34:15.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:15.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:15.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:16.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836488 2020-03-11T22:34:16.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488' 2020-03-11T22:34:16.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:16.038 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:16.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:34:16.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2020-03-11T22:34:16.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:16.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:16.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-11T22:34:16.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-60129542150' 2020-03-11T22:34:16.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:16.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836488 2020-03-11T22:34:16.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:16.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:16.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836488 2020-03-11T22:34:16.300 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:16.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836488 2020-03-11T22:34:16.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2020-03-11T22:34:16.301 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836488 2020-03-11T22:34:16.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:16.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836488 2020-03-11T22:34:16.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:17.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:17.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:18.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836488 2020-03-11T22:34:18.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:18.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:34:18.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:18.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:18.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:34:18.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:18.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:34:18.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:34:18.011 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:34:18.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:18.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672967 -lt 42949672967 2020-03-11T22:34:18.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:18.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-11T22:34:18.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:18.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:18.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-11T22:34:18.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:18.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-11T22:34:18.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-11T22:34:18.356 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 60129542150 2020-03-11T22:34:18.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:18.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542150 -lt 60129542150 2020-03-11T22:34:18.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:18.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:18.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:19.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:19.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:19.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:19.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:19.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:19.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:19.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:19.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:19.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:34:19.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:19.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:19.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:19.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:34:19.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:34:19.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:34:19.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:34:19.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:34:20.278 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:34:20.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:34:20.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:34:20.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:20.287 INFO:tasks.workunit.client.0.smithi130.stdout:primary and soon to be divergent is 1 2020-03-11T22:34:20.604 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:20.604 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:34:13.156281+0000 0'0 24:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:34:20.605 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:34:20.605 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:20.606 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:34:20.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:34:20.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:20.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:34:20.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:34:20.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:20.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:34:20.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:34:20.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:34:20.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:34:20.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:34:20.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:34:20.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:34:20.617 INFO:tasks.workunit.client.0.smithi130.stdout:writing initial objects 2020-03-11T22:34:20.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:34:20.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:34:20.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:34:20.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:34:20.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:34:20.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:34:20.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:20.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:34:21.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:21.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:34:21.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:21.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:34:21.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:21.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:34:21.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:21.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:34:21.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:21.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:34:22.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:22.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:34:22.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:22.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:34:22.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:22.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:34:22.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:22.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:34:22.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:22.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:34:22.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:22.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:34:23.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:34:23.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:34:23.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:34:23.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:34:23.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:34:23.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:34:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:34:24.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:34:24.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:34:24.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:34:24.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:34:24.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:34:24.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:34:24.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:34:24.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:34:24.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:34:24.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:34:24.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:34:24.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:34:24.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:34:24.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:34:24.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:34:24.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:34:24.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:24.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:34:25.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:34:25.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:34:25.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:34:25.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:34:25.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:34:25.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:34:25.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:34:25.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:34:25.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:34:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:34:25.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:34:25.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:34:25.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:34:25.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:34:25.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:34:25.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:25.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:34:26.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:34:26.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:34:26.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:34:26.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:34:26.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:34:26.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:34:26.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:34:26.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:34:26.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:34:26.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:34:26.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:34:26.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:34:26.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:34:26.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:34:26.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:34:26.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:34:26.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:26.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:34:27.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:34:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:34:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:34:27.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:34:27.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:34:27.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:34:27.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:34:27.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:34:27.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:34:27.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:34:27.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:34:27.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:34:27.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:34:27.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:34:27.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:34:27.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:34:27.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:34:27.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:27.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:34:28.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:28.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:34:28.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:28.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:34:28.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:28.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:34:28.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:28.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:34:28.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:28.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:34:28.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:34:28.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:34:28.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:34:28.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:34:28.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:28.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:28.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:28.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:28.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:34:28.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:28.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:28.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:28.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:28.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:28.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:28.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:28.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:28.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:28.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:28.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:28.789 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:34:28.789 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:34:28.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:28.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:28.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:28.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:34:28.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:34:28.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:28.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:29.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:34:29.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:34:29.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:29.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:29.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542154 2020-03-11T22:34:29.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-60129542154' 2020-03-11T22:34:29.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:29.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:34:29.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:29.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:29.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:34:29.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:29.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:34:29.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:34:29.183 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:34:29.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:29.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:34:29.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:30.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:30.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:30.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:34:30.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:30.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:34:30.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:30.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:30.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:34:30.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:30.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:34:30.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:34:30.916 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:34:30.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:31.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:34:31.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:31.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542154 2020-03-11T22:34:31.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:31.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542154 2020-03-11T22:34:31.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542154 2020-03-11T22:34:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2020-03-11T22:34:31.273 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 60129542154 2020-03-11T22:34:31.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:31.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542154 2020-03-11T22:34:31.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:31.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:31.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:32.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:32.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:32.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:32.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:32.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:32.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:32.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:32.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:32.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:34:32.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:32.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:32.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:34:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:34:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:34:32.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:34:32.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:34:32.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:34:32.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:34:32.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:34:33.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:34:33.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:34:33.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:33.214 INFO:tasks.workunit.client.0.smithi130.stdout:blackholing osds 0 2 2020-03-11T22:34:33.544 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:33.544 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:34:13.156281+0000 27'100 27:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:34:33.545 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:34:33.545 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:33.546 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:34:33.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:33.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:34:33.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:34:33.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:34:33.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:33.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:33.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:34:33.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:34:33.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:33.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:34:33.642 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:34:33.642 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:34:33.642 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:34:33.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:33.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:34:33.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:34:33.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:34:33.651 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:33.651 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:33.651 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:34:33.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:34:33.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:33.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:34:33.737 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:34:33.737 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:34:33.738 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:34:33.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:34:33.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:34:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:34:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:34:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:34:33.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:34:33.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:34:33.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:34:33.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:34:33.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:34:33.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:34:33.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:34:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:34:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:34:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:34:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:34:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:34:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:34:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:34:33.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:34:33.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:34:33.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:34:33.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:34:33.758 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:34:33.758 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:34:33.758 INFO:tasks.workunit.client.0.smithi130.stdout:writing divergent object 2020-03-11T22:34:33.759 INFO:tasks.workunit.client.0.smithi130.stdout:create missing divergent object 2020-03-11T22:34:34.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:34:34.098 INFO:tasks.workunit.client.0.smithi130.stderr:0 2020-03-11T22:34:34.098 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:34:34.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:34:34.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:34:34.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:34:34.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:34:34.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:34:34.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:34:34.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:34:34.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:34:34.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:34:34.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:34:34.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:34:34.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:34:34.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:34:34.104 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:34:34.105 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:34:34.105 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:34:34.105 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:34.105 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:34.106 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:34:34.106 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:34:34.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17424/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:34:34.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:34:34.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:34:34.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:34:34.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:34:34.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:34:34.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:34:34.198 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:34.198 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:34.198 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:34:34.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:34:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:34:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:34:34.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:34:34.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:34:44.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:34:44.310 INFO:tasks.workunit.client.0.smithi130.stderr:rados: no process found 2020-03-11T22:34:44.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:34:44.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:44.311 INFO:tasks.workunit.client.0.smithi130.stdout:killing all the osds 2020-03-11T22:34:44.641 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:44.641 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:34:34.585266+0000 27'101 27:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:34:44.641 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:34:44.641 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:44.642 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:34:44.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:34:44.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:34:44.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:34:44.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:34:44.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:34:44.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:34:44.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:34:44.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:44.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:34:45.296 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2020-03-11T22:34:45.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:45.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:34:45.803 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already down. 2020-03-11T22:34:45.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:34:45.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:34:46.296 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already down. 2020-03-11T22:34:46.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:46.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:34:46.799 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2020-03-11T22:34:46.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:34:47.289 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2020-03-11T22:34:47.297 INFO:tasks.workunit.client.0.smithi130.stdout:bringing up non_divergent 0 2 2020-03-11T22:34:47.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:34:47.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:47.618 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:47.618 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:34:34.585266+0000 27'101 27:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:34:47.618 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:34:47.619 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:47.620 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:34:47.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:47.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:34:47.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:47.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:47.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:34:47.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:47.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:34:47.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:47.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:47.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:47.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:47.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:34:47.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:47.633 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:47.634 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:34:47.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:34:47.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:34:47.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:47.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:47.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:47.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:47.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:34:47.636 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:34:47.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:34:47.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:47.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:47.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:47.657 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:47.653+0000 7f3156c35ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:47.658 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:47.655+0000 7f3156c35ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:47.660 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:47.657+0000 7f3156c35ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:47.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:34:47.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:47.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:34:47.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:47.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:47.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:47.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:47.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:47.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:34:47.976 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:34:48.175 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:48.171+0000 7f3156c35ec0 -1 Falling back to public interface 2020-03-11T22:34:48.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:49.033 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:49.030+0000 7f3156c35ec0 -1 osd.0 27 log_to_monitors {default=true} 2020-03-11T22:34:49.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:49.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:49.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:49.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:49.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:34:49.327 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:34:49.678 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up out weight 0 up_from 33 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/22541,v1:127.0.0.1:6803/22541] [v2:127.0.0.1:6804/22541,v1:127.0.0.1:6805/22541] exists,up f460e2f8-4c70-4627-a3ef-aff3188231e6 2020-03-11T22:34:49.678 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:34:49.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:49.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:49.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:49.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:49.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:34:49.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:49.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:49.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:34:49.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:49.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:34:49.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:49.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:49.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:49.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:49.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:34:49.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:34:49.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:49.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:49.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:49.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:49.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:49.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:49.681 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:49.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:49.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:34:49.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:34:49.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:34:49.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:49.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:49.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:49.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:49.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:49.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:49.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:49.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:49.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:49.703 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:49.700+0000 7ff6d8de0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:49.704 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:49.701+0000 7ff6d8de0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:49.706 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:49.703+0000 7ff6d8de0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:50.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:34:50.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:50.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:34:50.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:50.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:50.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:50.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:50.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:50.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:50.030 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:34:50.223 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:50.219+0000 7ff6d8de0ec0 -1 Falling back to public interface 2020-03-11T22:34:50.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:34:51.178 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:34:51.174+0000 7ff6d8de0ec0 -1 osd.2 27 log_to_monitors {default=true} 2020-03-11T22:34:51.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:34:51.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:51.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:34:51.403 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:34:51.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:51.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:51.754 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up out weight 0 up_from 34 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/22735,v1:127.0.0.1:6811/22735] [v2:127.0.0.1:6812/22735,v1:127.0.0.1:6813/22735] exists,up 8a6256f9-8a8b-47fd-8dc9-81d6ed0db004 2020-03-11T22:34:51.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:51.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:51.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:51.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:51.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:34:52.304 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already in. 2020-03-11T22:34:52.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:52.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:34:52.833 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2020-03-11T22:34:52.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:34:52.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:34:52.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:34:52.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:34:52.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:34:52.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:34:52.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:34:52.843 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:34:52.843 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:34:52.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:34:52.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:34:52.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:34:52.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:34:52.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:34:52.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:34:52.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:34:52.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:34:53.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:34:53.352 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:34:53.352 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:34:53.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:34:53.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:53.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:34:53.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:34:53.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771' 2020-03-11T22:34:53.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:53.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:34:53.573 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:34:53.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:34:53.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771 1-' 2020-03-11T22:34:53.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:34:53.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:34:53.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:34:53.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771 1- 2-146028888067' 2020-03-11T22:34:53.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:53.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-141733920771 2020-03-11T22:34:53.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:53.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:34:53.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-141733920771 2020-03-11T22:34:53.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:53.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:34:53.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2020-03-11T22:34:53.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:53.714 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 141733920771 2020-03-11T22:34:54.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 141733920771 2020-03-11T22:34:54.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:34:55.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:34:55.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:34:55.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920772 -lt 141733920771 2020-03-11T22:34:55.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:55.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:34:55.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:55.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:34:55.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:34:55.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:55.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:34:55.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:34:55.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:34:55.413 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 2020-03-11T22:34:55.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:34:55.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:34:55.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:34:55.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-146028888067 2020-03-11T22:34:55.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:34:55.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:34:55.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-146028888067 2020-03-11T22:34:55.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:34:55.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:34:55.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2020-03-11T22:34:55.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:34:55.758 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 146028888067 2020-03-11T22:34:56.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888067 -lt 146028888067 2020-03-11T22:34:56.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:34:56.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:56.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:56.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:34:56.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:56.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:56.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:56.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:56.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:56.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:56.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:56.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:56.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:56.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:56.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:57.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:57.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:34:57.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:34:57.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:34:57.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:57.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:34:57.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:57.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:57.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:57.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:57.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:57.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:57.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:57.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:57.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:57.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:57.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:57.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:58.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:58.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:58.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:58.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:58.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:58.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:58.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:58.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:58.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:34:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:34:58.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:58.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:58.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:58.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:58.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:58.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:58.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:58.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:59.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:59.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:59.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:59.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:59.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:59.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:59.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:59.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:59.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:59.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:59.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:59.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:00.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:00.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:00.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:35:00.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:00.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:35:00.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:00.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:00.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:00.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:00.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:00.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:00.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:00.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:00.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:00.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:00.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:00.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:01.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:01.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:01.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:01.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:01.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:01.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:01.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:01.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:01.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:01.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:01.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:01.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:35:01.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:01.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:35:02.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:02.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:02.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:02.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:02.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:02.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:02.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:02.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:02.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:02.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:02.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:02.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:03.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:03.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:03.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:03.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:03.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:03.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:03.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:03.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:03.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:03.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:35:03.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:03.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:35:05.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:05.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:05.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:05.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:05.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:05.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:05.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:05.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:05.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:05.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:05.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:05.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:06.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:06.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:06.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:06.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:06.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:06.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:06.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:06.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:06.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:06.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:35:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:35:09.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:09.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:09.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:09.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:09.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:09.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:09.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:09.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:10.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:10.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:10.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:10.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:10.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:10.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:10.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:10.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:11.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:11.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:11.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:35:11.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:11.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:35:17.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:17.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:17.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:17.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:17.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:17.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:17.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:17.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:17.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:17.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:18.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:18.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:18.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:18.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:18.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:18.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:35:18.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:35:18.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:35:25.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:35:25.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:25.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:25.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:25.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:25.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:25.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:25.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:26.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:35:26.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:26.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:26.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:26.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:35:26.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:35:26.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:35:26.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:35:26.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:35:26.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:35:26.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:35:26.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:35:26.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:35:27.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:35:27.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:35:27.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:35:27.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:35:27.655 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:35:27.655 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_fingerprint": "6e9f5683-af6f-4d4b-ab96-1b1b244064a2", 2020-03-11T22:35:27.655 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:35:27.655 INFO:tasks.workunit.client.0.smithi130.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:35:27.655 INFO:tasks.workunit.client.0.smithi130.stdout: "timestamp": "2020-03-11T22:35:27.651874+0000", 2020-03-11T22:35:27.656 INFO:tasks.workunit.client.0.smithi130.stdout: "tag": "", 2020-03-11T22:35:27.656 INFO:tasks.workunit.client.0.smithi130.stdout: "health": { 2020-03-11T22:35:27.656 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "HEALTH_WARN", 2020-03-11T22:35:27.656 INFO:tasks.workunit.client.0.smithi130.stdout: "checks": { 2020-03-11T22:35:27.656 INFO:tasks.workunit.client.0.smithi130.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:35:27.656 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:27.656 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:35:27.657 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:35:27.657 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 17 2020-03-11T22:35:27.657 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.657 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [], 2020-03-11T22:35:27.657 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:35:27.657 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.657 INFO:tasks.workunit.client.0.smithi130.stdout: "OSD_DOWN": { 2020-03-11T22:35:27.658 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:27.658 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:35:27.658 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 osds down", 2020-03-11T22:35:27.658 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:35:27.658 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.658 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:35:27.658 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.659 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "osd.1 (root=default,host=smithi130) is down" 2020-03-11T22:35:27.659 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.659 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.659 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:35:27.659 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.659 INFO:tasks.workunit.client.0.smithi130.stdout: "PG_DEGRADED": { 2020-03-11T22:35:27.659 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:35:27.660 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.661 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.661 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:35:27.661 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.661 INFO:tasks.workunit.client.0.smithi130.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:35:27.661 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:35:27.661 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:35:27.661 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:35:27.662 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:35:27.662 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.662 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:35:27.662 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.662 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:35:27.662 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.662 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.663 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:35:27.663 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.663 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.663 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:35:27.663 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.663 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.663 INFO:tasks.workunit.client.0.smithi130.stdout: "mutes": [] 2020-03-11T22:35:27.664 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.664 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_first_committed": 1, 2020-03-11T22:35:27.664 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_last_committed": 1, 2020-03-11T22:35:27.664 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap": { 2020-03-11T22:35:27.664 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:35:27.664 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:35:27.664 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:33:49.271031Z", 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:33:49.271031Z", 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release": 15, 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "features": { 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "persistent": [ 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "kraken", 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "luminous", 2020-03-11T22:35:27.665 INFO:tasks.workunit.client.0.smithi130.stdout: "mimic", 2020-03-11T22:35:27.666 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap-prune", 2020-03-11T22:35:27.666 INFO:tasks.workunit.client.0.smithi130.stdout: "nautilus", 2020-03-11T22:35:27.666 INFO:tasks.workunit.client.0.smithi130.stdout: "octopus" 2020-03-11T22:35:27.666 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.666 INFO:tasks.workunit.client.0.smithi130.stdout: "optional": [] 2020-03-11T22:35:27.666 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.666 INFO:tasks.workunit.client.0.smithi130.stdout: "mons": [ 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: "rank": 0, 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "a", 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.667 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:35:27.668 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 0 2020-03-11T22:35:27.668 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.668 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.668 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.668 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:35:27.668 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:35:27.668 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 0, 2020-03-11T22:35:27.669 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 0 2020-03-11T22:35:27.669 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.669 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.669 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.669 INFO:tasks.workunit.client.0.smithi130.stdout: "quorum": [ 2020-03-11T22:35:27.669 INFO:tasks.workunit.client.0.smithi130.stdout: 0 2020-03-11T22:35:27.669 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.670 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap": { 2020-03-11T22:35:27.670 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 39, 2020-03-11T22:35:27.670 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:35:27.670 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:33:50.214982+0000", 2020-03-11T22:35:27.670 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:34:52.916182+0000", 2020-03-11T22:35:27.670 INFO:tasks.workunit.client.0.smithi130.stdout: "last_up_change": "2020-03-11T22:34:51.255113+0000", 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "last_in_change": "2020-03-11T22:34:52.615969+0000", 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_num": 5800384, 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_set": [ 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "nodown", 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "noin", 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "noout", 2020-03-11T22:35:27.671 INFO:tasks.workunit.client.0.smithi130.stdout: "pglog_hardlimit", 2020-03-11T22:35:27.672 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snapdirs", 2020-03-11T22:35:27.672 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_deletes", 2020-03-11T22:35:27.672 INFO:tasks.workunit.client.0.smithi130.stdout: "sortbitwise" 2020-03-11T22:35:27.672 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.672 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_version": 7, 2020-03-11T22:35:27.672 INFO:tasks.workunit.client.0.smithi130.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:35:27.672 INFO:tasks.workunit.client.0.smithi130.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:35:27.673 INFO:tasks.workunit.client.0.smithi130.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:35:27.673 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_snapshot": "", 2020-03-11T22:35:27.673 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_max": 1, 2020-03-11T22:35:27.673 INFO:tasks.workunit.client.0.smithi130.stdout: "max_osd": 3, 2020-03-11T22:35:27.673 INFO:tasks.workunit.client.0.smithi130.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:35:27.673 INFO:tasks.workunit.client.0.smithi130.stdout: "min_compat_client": "jewel", 2020-03-11T22:35:27.673 INFO:tasks.workunit.client.0.smithi130.stdout: "require_osd_release": "octopus", 2020-03-11T22:35:27.674 INFO:tasks.workunit.client.0.smithi130.stdout: "pools": [ 2020-03-11T22:35:27.674 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.674 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:35:27.674 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_name": "test", 2020-03-11T22:35:27.674 INFO:tasks.workunit.client.0.smithi130.stdout: "create_time": "2020-03-11T22:34:11.062254+0000", 2020-03-11T22:35:27.674 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": 1, 2020-03-11T22:35:27.674 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_names": "hashpspool", 2020-03-11T22:35:27.675 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:35:27.675 INFO:tasks.workunit.client.0.smithi130.stdout: "size": 3, 2020-03-11T22:35:27.675 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 2, 2020-03-11T22:35:27.675 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_rule": 0, 2020-03-11T22:35:27.675 INFO:tasks.workunit.client.0.smithi130.stdout: "object_hash": 2, 2020-03-11T22:35:27.675 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:35:27.675 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num": 1, 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num": 1, 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num_target": 1, 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_target": 1, 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_pending": 1, 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pg_merge_meta": { 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "source_pgid": "0.0", 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "ready_epoch": 0, 2020-03-11T22:35:27.676 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 0, 2020-03-11T22:35:27.677 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 0, 2020-03-11T22:35:27.677 INFO:tasks.workunit.client.0.smithi130.stdout: "source_version": "0'0", 2020-03-11T22:35:27.677 INFO:tasks.workunit.client.0.smithi130.stdout: "target_version": "0'0" 2020-03-11T22:35:27.677 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.677 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "27", 2020-03-11T22:35:27.677 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend": "0", 2020-03-11T22:35:27.677 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:35:27.678 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:35:27.678 INFO:tasks.workunit.client.0.smithi130.stdout: "auid": 0, 2020-03-11T22:35:27.678 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:35:27.678 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_seq": 0, 2020-03-11T22:35:27.678 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_epoch": 0, 2020-03-11T22:35:27.678 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_snaps": [], 2020-03-11T22:35:27.678 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps": "[]", 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_bytes": 0, 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_objects": 0, 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "tiers": [], 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "tier_of": -1, 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "read_tier": -1, 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "write_tier": -1, 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_mode": "none", 2020-03-11T22:35:27.679 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_bytes": 0, 2020-03-11T22:35:27.680 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_objects": 0, 2020-03-11T22:35:27.680 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:35:27.680 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:35:27.680 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:35:27.680 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_flush_age": 0, 2020-03-11T22:35:27.680 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_evict_age": 0, 2020-03-11T22:35:27.680 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profile": "", 2020-03-11T22:35:27.681 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_params": { 2020-03-11T22:35:27.681 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "none" 2020-03-11T22:35:27.681 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.681 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_period": 0, 2020-03-11T22:35:27.681 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_count": 0, 2020-03-11T22:35:27.681 INFO:tasks.workunit.client.0.smithi130.stdout: "use_gmt_hitset": true, 2020-03-11T22:35:27.681 INFO:tasks.workunit.client.0.smithi130.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "grade_table": [], 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "stripe_width": 0, 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "expected_num_objects": 0, 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "fast_read": false, 2020-03-11T22:35:27.682 INFO:tasks.workunit.client.0.smithi130.stdout: "options": {}, 2020-03-11T22:35:27.683 INFO:tasks.workunit.client.0.smithi130.stdout: "application_metadata": {} 2020-03-11T22:35:27.683 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.683 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.683 INFO:tasks.workunit.client.0.smithi130.stdout: "osds": [ 2020-03-11T22:35:27.683 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.683 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:35:27.683 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "f460e2f8-4c70-4627-a3ef-aff3188231e6", 2020-03-11T22:35:27.684 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:35:27.684 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:35:27.684 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:35:27.684 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:35:27.684 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 5, 2020-03-11T22:35:27.684 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 27, 2020-03-11T22:35:27.684 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 33, 2020-03-11T22:35:27.685 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 38, 2020-03-11T22:35:27.685 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 28, 2020-03-11T22:35:27.685 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:35:27.685 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:35:27.685 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.685 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:35:27.686 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.687 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.687 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.687 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.687 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:35:27.687 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.687 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.687 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.688 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:35:27.688 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.688 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.688 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.688 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.688 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:35:27.688 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.689 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.689 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.689 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.689 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:27.689 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.689 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.689 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.690 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:35:27.690 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.690 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.690 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.690 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.690 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:35:27.690 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.691 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:35:27.692 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.692 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.692 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.692 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.692 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:35:27.692 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22541 2020-03-11T22:35:27.692 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.693 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.693 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.693 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6803/22541", 2020-03-11T22:35:27.693 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6805/22541", 2020-03-11T22:35:27.693 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6809/22541", 2020-03-11T22:35:27.693 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6807/22541", 2020-03-11T22:35:27.693 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:35:27.694 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:35:27.694 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:35:27.694 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.694 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.694 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.694 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:35:27.694 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "bec02fee-4bcc-4b9d-a58d-cfa0750b447c", 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 0, 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 0, 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 0, 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 10, 2020-03-11T22:35:27.695 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 21, 2020-03-11T22:35:27.696 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 29, 2020-03-11T22:35:27.696 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:35:27.696 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:35:27.696 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.696 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.696 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.696 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:35:27.697 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.697 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.697 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.697 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.697 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:35:27.697 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.697 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:35:27.698 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.699 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.699 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.699 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.699 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:35:27.699 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.699 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.699 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.700 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.700 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:27.700 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.700 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.700 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.700 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:35:27.700 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.701 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.701 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.701 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.701 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:35:27.701 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.702 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.702 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.702 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.702 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:27.702 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.702 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.702 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.703 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:35:27.703 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.703 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.703 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.703 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.703 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:35:27.703 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 18216 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/18216", 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/18216", 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18216", 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18216", 2020-03-11T22:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:35:27.705 INFO:tasks.workunit.client.0.smithi130.stdout: "exists" 2020-03-11T22:35:27.705 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.705 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.705 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.705 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:35:27.705 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "8a6256f9-8a8b-47fd-8dc9-81d6ed0db004", 2020-03-11T22:35:27.705 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:35:27.706 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:35:27.706 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:35:27.706 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:35:27.706 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 14, 2020-03-11T22:35:27.706 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 27, 2020-03-11T22:35:27.706 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 34, 2020-03-11T22:35:27.706 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 0, 2020-03-11T22:35:27.707 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 30, 2020-03-11T22:35:27.707 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:35:27.707 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:35:27.707 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.707 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.707 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.707 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:35:27.708 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.708 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.708 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.708 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.708 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:35:27.708 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.708 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.709 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.709 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.709 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:35:27.709 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.709 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.709 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.709 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:35:27.710 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.710 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.710 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.710 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.710 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:35:27.710 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.710 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:35:27.711 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.712 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.712 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.712 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.712 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:35:27.712 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.712 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.712 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.713 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.713 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:35:27.713 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:35:27.713 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.713 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:35:27.713 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:35:27.713 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 22735 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.714 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.715 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/22735", 2020-03-11T22:35:27.715 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/22735", 2020-03-11T22:35:27.715 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22735", 2020-03-11T22:35:27.715 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22735", 2020-03-11T22:35:27.715 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:35:27.715 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:35:27.715 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:35:27.716 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.716 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.716 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.716 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_xinfo": [ 2020-03-11T22:35:27.716 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.716 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:35:27.716 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:34:45.055907+0000", 2020-03-11T22:35:27.717 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:35:27.717 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:35:27.717 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:35:27.717 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:35:27.717 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:56.796028+0000", 2020-03-11T22:35:27.717 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:35:27.717 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:34:45.615871+0000", 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:35:27.718 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:34:02.914730+0000", 2020-03-11T22:35:27.719 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:35:27.719 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.719 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.719 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:35:27.719 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:34:46.106119+0000", 2020-03-11T22:35:27.719 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:35:27.719 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:35:27.720 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:35:27.720 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:35:27.720 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:34:09.038672+0000", 2020-03-11T22:35:27.720 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:35:27.720 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.720 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.720 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap": [], 2020-03-11T22:35:27.721 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap_items": [], 2020-03-11T22:35:27.721 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_temp": [], 2020-03-11T22:35:27.721 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_temp": [], 2020-03-11T22:35:27.721 INFO:tasks.workunit.client.0.smithi130.stdout: "blacklist": {}, 2020-03-11T22:35:27.721 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profiles": { 2020-03-11T22:35:27.721 INFO:tasks.workunit.client.0.smithi130.stdout: "default": { 2020-03-11T22:35:27.721 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "2", 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van" 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps_queue": [], 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: "new_removed_snaps": [], 2020-03-11T22:35:27.722 INFO:tasks.workunit.client.0.smithi130.stdout: "new_purged_snaps": [], 2020-03-11T22:35:27.723 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_node_flags": {}, 2020-03-11T22:35:27.723 INFO:tasks.workunit.client.0.smithi130.stdout: "device_class_flags": {} 2020-03-11T22:35:27.723 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.723 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_metadata": [ 2020-03-11T22:35:27.723 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.723 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:35:27.723 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:35:27.724 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6804/22541,v1:127.0.0.1:6805/22541]", 2020-03-11T22:35:27.724 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:35:27.724 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:35:27.724 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:27.724 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:27.724 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:27.724 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:27.725 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:27.726 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:35:27.726 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:27.726 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:27.726 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:35:27.726 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:35:27.726 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:35:27.726 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:27.727 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:35:27.727 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:35:27.727 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:27.727 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:35:27.727 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6802/22541,v1:127.0.0.1:6803/22541]", 2020-03-11T22:35:27.727 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:35:27.727 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/22541,v1:127.0.0.1:6809/22541]", 2020-03-11T22:35:27.728 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/22541,v1:127.0.0.1:6807/22541]", 2020-03-11T22:35:27.728 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:35:27.728 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:35:27.728 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:27.728 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:27.729 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:27.729 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:35:27.729 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:27.729 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:27.729 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:35:27.729 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:35:27.729 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:27.730 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:35:27.730 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.730 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.730 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:35:27.730 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:35:27.730 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/18216,v1:127.0.0.1:6813/18216]", 2020-03-11T22:35:27.730 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:27.731 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:27.732 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:35:27.732 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:27.732 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:27.732 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:27.732 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:27.732 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:35:27.733 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:27.733 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:27.733 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:35:27.733 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:35:27.733 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:35:27.733 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:27.733 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:35:27.734 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:35:27.734 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:27.734 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:35:27.734 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/18216,v1:127.0.0.1:6811/18216]", 2020-03-11T22:35:27.734 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:35:27.734 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18216,v1:127.0.0.1:6817/18216]", 2020-03-11T22:35:27.734 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18216,v1:127.0.0.1:6815/18216]", 2020-03-11T22:35:27.735 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:35:27.735 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:35:27.735 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:27.735 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:27.735 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:27.735 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:35:27.735 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.736 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:35:27.737 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:35:27.737 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/22735,v1:127.0.0.1:6813/22735]", 2020-03-11T22:35:27.737 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:35:27.737 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:35:27.737 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:35:27.737 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:35:27.738 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:35:27.739 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:35:27.739 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:35:27.739 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:35:27.739 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:35:27.739 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:35:27.739 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:35:27.739 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/22735,v1:127.0.0.1:6811/22735]", 2020-03-11T22:35:27.740 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:35:27.741 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22735,v1:127.0.0.1:6817/22735]", 2020-03-11T22:35:27.741 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22735,v1:127.0.0.1:6815/22735]", 2020-03-11T22:35:27.741 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:35:27.741 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:35:27.741 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:35:27.741 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:35:27.741 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:35:27.742 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:35:27.742 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:35:27.742 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:35:27.742 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:35:27.742 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:35:27.742 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:35:27.742 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:35:27.743 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.743 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.743 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_first_committed": 1, 2020-03-11T22:35:27.743 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_last_committed": 39, 2020-03-11T22:35:27.743 INFO:tasks.workunit.client.0.smithi130.stdout: "crushmap": { 2020-03-11T22:35:27.743 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": [ 2020-03-11T22:35:27.743 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.0", 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.1", 2020-03-11T22:35:27.744 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:35:27.745 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.745 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.745 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:35:27.745 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.2", 2020-03-11T22:35:27.745 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:35:27.745 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.745 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: "types": [ 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 0, 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd" 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:35:27.746 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "host" 2020-03-11T22:35:27.747 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.747 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.747 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 2, 2020-03-11T22:35:27.747 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "chassis" 2020-03-11T22:35:27.747 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.747 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.747 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 3, 2020-03-11T22:35:27.748 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "rack" 2020-03-11T22:35:27.748 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.748 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.748 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 4, 2020-03-11T22:35:27.748 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "row" 2020-03-11T22:35:27.748 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.748 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.749 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 5, 2020-03-11T22:35:27.749 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pdu" 2020-03-11T22:35:27.749 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.749 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.749 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 6, 2020-03-11T22:35:27.749 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pod" 2020-03-11T22:35:27.749 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 7, 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "room" 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 8, 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "datacenter" 2020-03-11T22:35:27.750 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.751 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.751 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 9, 2020-03-11T22:35:27.751 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "zone" 2020-03-11T22:35:27.751 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.751 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.751 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 10, 2020-03-11T22:35:27.751 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "region" 2020-03-11T22:35:27.752 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.752 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.752 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:35:27.752 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "root" 2020-03-11T22:35:27.752 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.752 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.752 INFO:tasks.workunit.client.0.smithi130.stdout: "buckets": [ 2020-03-11T22:35:27.753 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.753 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -1, 2020-03-11T22:35:27.753 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default", 2020-03-11T22:35:27.753 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:35:27.753 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:35:27.753 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:35:27.753 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:35:27.754 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:35:27.754 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:35:27.754 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.754 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:35:27.754 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:35:27.754 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:35:27.754 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.755 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.755 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.755 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.755 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -2, 2020-03-11T22:35:27.755 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default~hdd", 2020-03-11T22:35:27.755 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:35:27.755 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:35:27.756 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:35:27.756 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:35:27.756 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:35:27.756 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:35:27.756 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.756 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:35:27.756 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:35:27.757 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:35:27.757 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.757 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.757 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.757 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.757 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:35:27.757 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130", 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.758 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:35:27.759 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:35:27.759 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:35:27.759 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.759 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.759 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:35:27.759 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:35:27.759 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.760 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.761 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.761 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:35:27.761 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130~hdd", 2020-03-11T22:35:27.761 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:35:27.761 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:35:27.761 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:35:27.761 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:35:27.762 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:35:27.762 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:35:27.762 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.762 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:35:27.762 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:35:27.762 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:35:27.762 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.763 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.763 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:35:27.763 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:35:27.763 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:35:27.763 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.763 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.763 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:35:27.764 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:35:27.764 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:35:27.764 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.764 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.764 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.764 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.764 INFO:tasks.workunit.client.0.smithi130.stdout: "rules": [ 2020-03-11T22:35:27.765 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.765 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 0, 2020-03-11T22:35:27.765 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "replicated_rule", 2020-03-11T22:35:27.765 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 0, 2020-03-11T22:35:27.765 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:35:27.765 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 1, 2020-03-11T22:35:27.765 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 10, 2020-03-11T22:35:27.766 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:35:27.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.766 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:35:27.766 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:35:27.766 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:35:27.766 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_firstn", 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.767 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:35:27.768 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.768 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.768 INFO:tasks.workunit.client.0.smithi130.stdout: "tunables": { 2020-03-11T22:35:27.768 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_tries": 0, 2020-03-11T22:35:27.768 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:35:27.768 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_total_tries": 50, 2020-03-11T22:35:27.768 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:35:27.769 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:35:27.769 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_stable": 1, 2020-03-11T22:35:27.769 INFO:tasks.workunit.client.0.smithi130.stdout: "straw_calc_version": 1, 2020-03-11T22:35:27.769 INFO:tasks.workunit.client.0.smithi130.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:35:27.769 INFO:tasks.workunit.client.0.smithi130.stdout: "profile": "jewel", 2020-03-11T22:35:27.769 INFO:tasks.workunit.client.0.smithi130.stdout: "optimal_tunables": 1, 2020-03-11T22:35:27.769 INFO:tasks.workunit.client.0.smithi130.stdout: "legacy_tunables": 0, 2020-03-11T22:35:27.770 INFO:tasks.workunit.client.0.smithi130.stdout: "minimum_required_version": "jewel", 2020-03-11T22:35:27.770 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables": 1, 2020-03-11T22:35:27.770 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables2": 1, 2020-03-11T22:35:27.770 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v2_rules": 0, 2020-03-11T22:35:27.770 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables3": 1, 2020-03-11T22:35:27.770 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v3_rules": 0, 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v4_buckets": 1, 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables5": 1, 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v5_rules": 0 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_args": {} 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: "fsmap": { 2020-03-11T22:35:27.771 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:35:27.772 INFO:tasks.workunit.client.0.smithi130.stdout: "default_fscid": -1, 2020-03-11T22:35:27.772 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": { 2020-03-11T22:35:27.772 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": {}, 2020-03-11T22:35:27.772 INFO:tasks.workunit.client.0.smithi130.stdout: "ro_compat": {}, 2020-03-11T22:35:27.772 INFO:tasks.workunit.client.0.smithi130.stdout: "incompat": { 2020-03-11T22:35:27.772 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_1": "base v0.20", 2020-03-11T22:35:27.772 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:35:27.773 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:35:27.773 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:35:27.773 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:35:27.773 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:35:27.773 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_8": "no anchor table", 2020-03-11T22:35:27.773 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_9": "file layout v2", 2020-03-11T22:35:27.773 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:35:27.774 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.774 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.774 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_flags": { 2020-03-11T22:35:27.774 INFO:tasks.workunit.client.0.smithi130.stdout: "enable_multiple": false, 2020-03-11T22:35:27.774 INFO:tasks.workunit.client.0.smithi130.stdout: "ever_enabled_multiple": false 2020-03-11T22:35:27.774 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.774 INFO:tasks.workunit.client.0.smithi130.stdout: "standbys": [], 2020-03-11T22:35:27.775 INFO:tasks.workunit.client.0.smithi130.stdout: "filesystems": [] 2020-03-11T22:35:27.775 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.775 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:35:27.775 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:35:27.775 INFO:tasks.workunit.client.0.smithi130.stdout: "auth": { 2020-03-11T22:35:27.775 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:35:27.775 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 5, 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: "num_secrets": 3 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_active": 1, 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_unknown": 0, 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osd": 3, 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_sum": { 2020-03-11T22:35:27.776 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:35:27.777 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48300, 2020-03-11T22:35:27.777 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:35:27.777 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:35:27.777 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:35:27.777 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:35:27.777 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:35:27.777 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:35:27.778 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:35:27.778 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:35:27.778 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:35:27.778 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:35:27.778 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:35:27.778 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:35:27.778 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 100, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 100, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:35:27.779 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:35:27.780 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:35:27.780 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:35:27.780 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:35:27.780 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:35:27.780 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:35:27.780 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:35:27.780 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:35:27.781 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:35:27.781 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:35:27.781 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:35:27.781 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:35:27.781 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:35:27.781 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:35:27.781 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:35:27.782 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:35:27.783 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:35:27.783 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 0, 2020-03-11T22:35:27.783 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 0, 2020-03-11T22:35:27.783 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:35:27.783 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:27.783 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:35:27.783 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:35:27.784 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:35:27.784 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.784 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 16, 2020-03-11T22:35:27.784 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 16, 2020-03-11T22:35:27.784 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 2, 2020-03-11T22:35:27.784 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 2, 2020-03-11T22:35:27.784 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 0 2020-03-11T22:35:27.785 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.785 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum": { 2020-03-11T22:35:27.785 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:35:27.785 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:35:27.785 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:35:27.785 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:35:27.785 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:35:27.786 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:35:27.786 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:35:27.786 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3178944, 2020-03-11T22:35:27.786 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 32960, 2020-03-11T22:35:27.786 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 14, 2020-03-11T22:35:27.787 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145713, 2020-03-11T22:35:27.787 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393856, 2020-03-11T22:35:27.787 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:35:27.787 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:35:27.787 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318867308544, 2020-03-11T22:35:27.787 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:35:27.787 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 33751040, 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 403151, 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 15180, 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221210292 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.788 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:35:27.789 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:35:27.789 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:35:27.789 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 1, 2020-03-11T22:35:27.789 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:35:27.789 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:35:27.789 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:35:27.789 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.790 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:35:27.790 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:35:27.790 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:35:27.790 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:35:27.790 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:35:27.790 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.790 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:35:27.791 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:35:27.791 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.791 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum_by_class": { 2020-03-11T22:35:27.791 INFO:tasks.workunit.client.0.smithi130.stdout: "hdd": { 2020-03-11T22:35:27.791 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:35:27.791 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:35:27.791 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3178944, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 32960, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 14, 2020-03-11T22:35:27.792 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145713, 2020-03-11T22:35:27.793 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393856, 2020-03-11T22:35:27.793 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:35:27.793 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:35:27.793 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318867308544, 2020-03-11T22:35:27.793 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:35:27.793 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 33751040, 2020-03-11T22:35:27.793 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 403151, 2020-03-11T22:35:27.794 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:35:27.794 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:27.794 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:35:27.794 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 15180, 2020-03-11T22:35:27.794 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221210292 2020-03-11T22:35:27.794 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.794 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:35:27.795 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:35:27.795 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:35:27.795 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 1, 2020-03-11T22:35:27.795 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:35:27.795 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:35:27.795 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:35:27.795 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.796 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:35:27.796 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:35:27.796 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:35:27.796 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:35:27.796 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:35:27.796 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.796 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:35:27.797 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:35:27.797 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.797 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.797 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_stats": [ 2020-03-11T22:35:27.797 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.797 INFO:tasks.workunit.client.0.smithi130.stdout: "poolid": 1, 2020-03-11T22:35:27.797 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:35:27.798 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:35:27.798 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48300, 2020-03-11T22:35:27.798 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:35:27.798 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:35:27.798 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:35:27.798 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:35:27.798 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:35:27.799 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 100, 2020-03-11T22:35:27.800 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 100, 2020-03-11T22:35:27.800 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:35:27.800 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:35:27.800 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:35:27.800 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:35:27.800 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:35:27.801 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:35:27.801 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:35:27.801 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:35:27.801 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:35:27.801 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:35:27.802 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:35:27.802 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:35:27.802 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:35:27.802 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:35:27.802 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:35:27.802 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:35:27.802 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:35:27.803 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:35:27.803 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:35:27.803 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:35:27.803 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:35:27.803 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:35:27.803 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:35:27.803 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 19660800, 2020-03-11T22:35:27.804 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 144900, 2020-03-11T22:35:27.805 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:35:27.805 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:35:27.805 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:35:27.805 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:35:27.805 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:35:27.805 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.805 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 16, 2020-03-11T22:35:27.806 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 16, 2020-03-11T22:35:27.806 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 2, 2020-03-11T22:35:27.806 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 2, 2020-03-11T22:35:27.806 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 3 2020-03-11T22:35:27.806 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.806 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.806 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_stats": [ 2020-03-11T22:35:27.807 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.807 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:35:27.807 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 141733920778 2020-03-11T22:35:27.807 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.807 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.807 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 42949672975 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 146028888074 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.808 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_state": [ 2020-03-11T22:35:27.809 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.809 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2020-03-11T22:35:27.809 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 1 2020-03-11T22:35:27.809 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.809 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.809 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_osd": [ 2020-03-11T22:35:27.809 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:35:27.810 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:35:27.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 0, 2020-03-11T22:35:27.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:27.811 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.811 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.811 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:35:27.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 1, 2020-03-11T22:35:27.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:35:27.812 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:35:27.812 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.812 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.812 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [ 2020-03-11T22:35:27.812 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:35:27.812 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:35:27.812 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": {} 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: "servicemap": { 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:33:50.215229+0000", 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: "services": {} 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:35:27.813 INFO:tasks.workunit.client.0.smithi130.stdout: "paxos": { 2020-03-11T22:35:27.814 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:35:27.814 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 302, 2020-03-11T22:35:27.814 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pn": 0, 2020-03-11T22:35:27.814 INFO:tasks.workunit.client.0.smithi130.stdout: "accepted_pn": 0 2020-03-11T22:35:27.814 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:35:27.814 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:35:27.815 INFO:tasks.workunit.client.0.smithi130.stderr:report 3125624121 2020-03-11T22:35:27.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:35:27.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:35:27.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:35:27.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:35:27.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:27.816 INFO:tasks.workunit.client.0.smithi130.stdout:writing non-divergent object existing_4 2020-03-11T22:35:27.986 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:27.986 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:35:08.479158+0000 27'100 39:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:27.986 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:27.986 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:27.988 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:27.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:35:28.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:35:28.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:28.037 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery 2020-03-11T22:35:28.357 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:28.357 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:35:08.479158+0000 27'100 39:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:28.357 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:28.357 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:28.359 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:28.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:28.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:35:28.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:35:28.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:35:28.368 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:28.368 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:28.369 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:35:28.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:35:28.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:28.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:35:28.459 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:28.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:35:28.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:35:28.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:35:28.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:28.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:28.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:35:28.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:35:28.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:28.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:35:28.559 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:28.566 INFO:tasks.workunit.client.0.smithi130.stdout:revive divergent 1 2020-03-11T22:35:28.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:35:28.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:28.908 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:28.908 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:35:08.479158+0000 27'100 39:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:28.909 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:28.909 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:28.910 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:28.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:35:29.538 INFO:tasks.workunit.client.0.smithi130.stderr:noup is set 2020-03-11T22:35:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:35:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:35:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:35:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:35:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:35:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:29.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:29.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:29.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:29.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:29.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:29.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:29.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:29.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:35:29.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:29.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:35:29.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:29.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:29.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:29.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:29.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:35:29.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:35:29.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:35:29.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:29.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:29.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:29.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:29.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:29.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:29.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:35:29.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:35:29.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:29.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:35:29.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:29.554 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:35:29.554 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:35:29.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:35:29.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:35:29.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:35:29.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:35:29.575 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:29.572+0000 7f6949d18ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:29.577 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:29.574+0000 7f6949d18ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:29.578 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:29.575+0000 7f6949d18ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:29.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:35:29.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:35:30.095 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:30.091+0000 7f6949d18ec0 -1 Falling back to public interface 2020-03-11T22:35:31.024 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:31.021+0000 7f6949d18ec0 -1 osd.1 27 log_to_monitors {default=true} 2020-03-11T22:35:34.897 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery divergent 2020-03-11T22:35:34.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:35:34.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:35.228 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:35.228 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:35:08.479158+0000 39'101 39:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:35.228 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:35.229 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:35.229 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:35.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:35:35.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:35:35.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:35:35.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:35.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:35.240 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:35:35.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:35:35.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:35.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:35:35.330 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:35:35.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:35:35.934 INFO:tasks.workunit.client.0.smithi130.stderr:noup is unset 2020-03-11T22:35:35.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:35:35.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:35.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:35:35.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:35.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:35.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:35.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:35.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:35.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:35.944 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:35:36.315 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 33 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/22541,v1:127.0.0.1:6803/22541] [v2:127.0.0.1:6804/22541,v1:127.0.0.1:6805/22541] exists,up f460e2f8-4c70-4627-a3ef-aff3188231e6 2020-03-11T22:35:36.316 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:35:36.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:35:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:36.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:36.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:36.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:36.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:36.655 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/24203,v1:127.0.0.1:6819/24203] [v2:127.0.0.1:6820/24203,v1:127.0.0.1:6821/24203] exists,up bec02fee-4bcc-4b9d-a58d-cfa0750b447c 2020-03-11T22:35:36.655 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:35:36.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:36.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:36.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:36.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:35:36.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:36.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:35:36.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:36.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:36.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:36.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:36.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:36.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:37.024 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/22735,v1:127.0.0.1:6811/22735] [v2:127.0.0.1:6812/22735,v1:127.0.0.1:6813/22735] exists,up 8a6256f9-8a8b-47fd-8dc9-81d6ed0db004 2020-03-11T22:35:37.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:37.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:37.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:37.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:37.345 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:37.345 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2020-03-11T22:35:35.964071+0000 27'101 43:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:37.345 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:37.345 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:37.346 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:37.354 INFO:tasks.workunit.client.0.smithi130.stdout:wait for peering 2020-03-11T22:35:37.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:35:37.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:37.671 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:37.672 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2020-03-11T22:35:35.964071+0000 27'101 43:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:37.672 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:37.672 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:37.673 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:37.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:35:49.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:35:49.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:49.186 INFO:tasks.workunit.client.0.smithi130.stdout:killing divergent 1 2020-03-11T22:35:49.517 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:49.517 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:35:36.143344+0000 39'101 44:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:49.517 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:49.518 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:49.519 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:49.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:35:49.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:49.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:49.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:49.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:49.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:49.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:49.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:35:49.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:49.633 INFO:tasks.workunit.client.0.smithi130.stdout:reviving divergent 1 2020-03-11T22:35:49.951 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:49.952 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:35:36.143344+0000 39'101 44:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:35:49.952 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:35:49.952 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:49.953 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:35:49.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:35:49.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:35:49.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:35:49.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:35:49.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:35:49.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:49.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:49.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:49.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:49.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:49.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:49.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:49.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:49.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:35:49.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:49.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:35:49.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:49.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:49.970 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:49.970 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:49.970 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:35:49.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:35:49.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:35:49.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:49.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:49.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:35:49.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:35:49.974 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:35:49.991 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:49.988+0000 7f9a0a05bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:49.993 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:49.989+0000 7f9a0a05bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:49.994 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:49.991+0000 7f9a0a05bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:50.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:35:50.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:50.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:50.312 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:35:50.517 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:50.513+0000 7f9a0a05bec0 -1 Falling back to public interface 2020-03-11T22:35:50.686 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/24203,v1:127.0.0.1:6819/24203] [v2:127.0.0.1:6820/24203,v1:127.0.0.1:6821/24203] exists,up bec02fee-4bcc-4b9d-a58d-cfa0750b447c 2020-03-11T22:35:50.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:50.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:50.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:50.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:35:51.383 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:35:51.380+0000 7f9a0a05bec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-11T22:36:10.687 INFO:tasks.workunit.client.0.smithi130.stdout:allowing recovery 2020-03-11T22:36:10.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:36:10.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:11.016 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:11.016 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:35:52.561496+0000 44'102 47:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:36:11.016 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:36:11.016 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:11.017 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:36:11.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:36:11.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:36:11.146 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:36:11.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:36:11.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:36:11.275 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:36:11.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:36:11.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:36:11.404 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:36:11.412 INFO:tasks.workunit.client.0.smithi130.stdout:reading divergent objects 2020-03-11T22:36:11.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:36:11.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:11.729 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:11.730 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:35:52.561496+0000 44'102 47:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:34:11.144307+0000 0'0 2020-03-11T22:34:11.144307+0000 0 2020-03-11T22:36:11.730 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:36:11.730 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:11.731 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:36:11.740 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:36:11.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:36:11.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:11.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:36:11.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:11.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:36:11.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:11.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:36:11.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:36:11.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:36:11.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:36:11.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:36:11.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:36:11.871 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4524.0:1 2020-03-11T22:34:33.770828+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:36:11.871 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4524.0:1 2020-03-11T22:34:33.770828+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:36:11.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:36:11.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'101 2020-03-11T22:36:11.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:36:11.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:36:11.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 27'101 2020-03-11T22:36:11.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4524.0:1 2020-03-11T22:34:33.770828+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:36:11.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:35:36.106+0000 7f6927ed3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:36:11.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:36:11.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:36:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:36:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:36:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:36:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:36:11.876 INFO:tasks.workunit.client.0.smithi130.stdout:success 2020-03-11T22:36:12.338 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' does not exist 2020-03-11T22:36:12.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:36:12.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:12.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:12.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:12.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:12.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:12.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:12.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:36:12.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:36:12.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:36:12.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:36:12.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:12.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:12.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:12.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:12.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:12.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:12.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:36:12.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:36:12.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:36:12.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:36:12.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:36:12.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:36:12.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:12.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:36:12.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:36:12.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:12.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:36:12.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:36:12.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:36:12.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:36:12.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:12.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:12.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:36:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:36:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:36:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:36:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:36:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:36:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:12.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:12.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:36:12.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:36:12.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:36:12.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:36:12.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:36:12.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:36:12.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:12.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:36:12.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:36:12.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:12.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:36:12.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:36:12.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:36:12.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:36:12.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:12.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:36:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:36:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:36:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:36:12.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:36:12.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:12.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17424 2020-03-11T22:36:12.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:36:12.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:36:12.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:36:12.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:36:12.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:36:12.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:36:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:36:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:36:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:36:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:36:12.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:36:12.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:36:12.492 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:36:12.492 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:36:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:36:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:36:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:36:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:36:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:36:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:36:12.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:36:12.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:36:12.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:12.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:12.809 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:12.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:12.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:36:12.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:36:12.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:36:12.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:36:12.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:36:12.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:12.973 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:36:12.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get fsid 2020-03-11T22:36:13.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:36:13.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:36:13.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:36:13.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:36:13.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:36:13.060 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:36:13.060 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:36:13.060 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:36:13.060 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:13.060 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:13.061 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:13.061 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:36:13.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:36:13.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get mon_host 2020-03-11T22:36:13.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:36:13.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:36:13.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:36:13.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:36:13.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:36:13.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:36:13.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:36:14.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:36:14.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:14.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:14.012 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:14.012 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:14.012 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:14.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:14.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:36:14.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:36:14.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:14.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:36:14.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:14.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:14.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:36:14.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:14.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:36:14.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:14.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:14.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:14.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:14.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:36:14.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:36:14.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:14.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:36:14.044 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:14.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1b8f82d0-d42b-4d93-ab95-f22ad2bc035e 2020-03-11T22:36:14.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 1b8f82d0-d42b-4d93-ab95-f22ad2bc035e' 2020-03-11T22:36:14.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:14.046 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 1b8f82d0-d42b-4d93-ab95-f22ad2bc035e 2020-03-11T22:36:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDeZ2leKXpfAxAADTK947iv2WActbgZlnV8fQ== 2020-03-11T22:36:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDeZ2leKXpfAxAADTK947iv2WActbgZlnV8fQ=="}' 2020-03-11T22:36:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1b8f82d0-d42b-4d93-ab95-f22ad2bc035e -i td/divergent-priors/0/new.json 2020-03-11T22:36:14.361 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:36:14.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:36:14.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDeZ2leKXpfAxAADTK947iv2WActbgZlnV8fQ== --osd-uuid 1b8f82d0-d42b-4d93-ab95-f22ad2bc035e 2020-03-11T22:36:14.399 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:14.397+0000 7f19a8239ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:14.401 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:14.399+0000 7f19a8239ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:14.402 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:14.400+0000 7f19a8239ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:14.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:14.400+0000 7f19a8239ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:14.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:14.400+0000 7f19a8239ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:14.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:14.400+0000 7f19a8239ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:14.469 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:14.467+0000 7f19a8239ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:36:17.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:36:17.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:17.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:36:17.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:17.424 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:36:17.871 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:36:17.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:36:17.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:17.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:17.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:17.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:17.898 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:17.896+0000 7f1520a55ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:17.900 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:17.898+0000 7f1520a55ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:17.901 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:17.899+0000 7f1520a55ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:36:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:18.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:18.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:18.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:18.220 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:36:18.422 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:18.419+0000 7f1520a55ec0 -1 Falling back to public interface 2020-03-11T22:36:18.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:19.185 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:19.182+0000 7f1520a55ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:36:19.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:19.565 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:36:19.924 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25505,v1:127.0.0.1:6803/25505] [v2:127.0.0.1:6804/25505,v1:127.0.0.1:6805/25505] exists,up 1b8f82d0-d42b-4d93-ab95-f22ad2bc035e 2020-03-11T22:36:19.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:19.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:19.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:19.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:19.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:36:19.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:19.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:19.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:36:19.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:19.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:36:19.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:19.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:19.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:19.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:19.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:36:19.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:36:19.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:19.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:19.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:19.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:19.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:19.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:19.927 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:19.928 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:19.928 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:19.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:19.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:19.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:19.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:19.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:19.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:19.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:19.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:19.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:19.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:19.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:19.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:19.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:36:19.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:19.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e938a584-5c23-4b9f-8a94-a38336bc98c0 2020-03-11T22:36:19.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 e938a584-5c23-4b9f-8a94-a38336bc98c0' 2020-03-11T22:36:19.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:19.931 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 e938a584-5c23-4b9f-8a94-a38336bc98c0 2020-03-11T22:36:19.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDjZ2le/mcMOBAAPVHwKnWhgo9/AlFlvKIXvQ== 2020-03-11T22:36:19.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDjZ2le/mcMOBAAPVHwKnWhgo9/AlFlvKIXvQ=="}' 2020-03-11T22:36:19.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e938a584-5c23-4b9f-8a94-a38336bc98c0 -i td/divergent-priors/1/new.json 2020-03-11T22:36:20.533 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:36:20.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:36:20.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDjZ2le/mcMOBAAPVHwKnWhgo9/AlFlvKIXvQ== --osd-uuid e938a584-5c23-4b9f-8a94-a38336bc98c0 2020-03-11T22:36:20.572 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:20.569+0000 7f015ecbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:20.584 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:20.581+0000 7f015ecbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:20.585 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:20.583+0000 7f015ecbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:20.586 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:20.583+0000 7f015ecbfec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:20.586 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:20.584+0000 7f015ecbfec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:20.586 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:20.584+0000 7f015ecbfec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:36:20.642 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:20.639+0000 7f015ecbfec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:36:23.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:36:23.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:23.651 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:36:23.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:36:23.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:24.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:36:24.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:24.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:24.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:24.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:24.114 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:36:24.140 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:24.136+0000 7fcf66750ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:24.141 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:24.138+0000 7fcf66750ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:24.143 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:24.139+0000 7fcf66750ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:36:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:36:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:24.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:24.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:36:24.461 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:36:24.660 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:24.657+0000 7fcf66750ec0 -1 Falling back to public interface 2020-03-11T22:36:24.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:25.490 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:25.487+0000 7fcf66750ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:36:25.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:25.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:25.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:25.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:25.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:36:25.805 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:36:26.185 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25860,v1:127.0.0.1:6811/25860] [v2:127.0.0.1:6812/25860,v1:127.0.0.1:6813/25860] exists,up e938a584-5c23-4b9f-8a94-a38336bc98c0 2020-03-11T22:36:26.185 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 ad75aea1-426e-4305-9d25-2742006cffd6 2020-03-11T22:36:26.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:26.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:26.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:26.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:36:26.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:36:26.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:26.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:26.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:36:26.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:26.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:36:26.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:26.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:26.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:26.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:26.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:36:26.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:36:26.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:26.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:26.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:26.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:26.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:26.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:26.189 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:26.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:26.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:26.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:26.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:36:26.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:26.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:26.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:26.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:26.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:26.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:26.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:26.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:26.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:26.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:26.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:36:26.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:26.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ad75aea1-426e-4305-9d25-2742006cffd6 2020-03-11T22:36:26.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 ad75aea1-426e-4305-9d25-2742006cffd6' 2020-03-11T22:36:26.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:26.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDqZ2le6THrCxAA3lC6G5lgHDLDw8hZBjRi3A== 2020-03-11T22:36:26.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDqZ2le6THrCxAA3lC6G5lgHDLDw8hZBjRi3A=="}' 2020-03-11T22:36:26.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ad75aea1-426e-4305-9d25-2742006cffd6 -i td/divergent-priors/2/new.json 2020-03-11T22:36:26.713 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:36:26.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:36:26.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDqZ2le6THrCxAA3lC6G5lgHDLDw8hZBjRi3A== --osd-uuid ad75aea1-426e-4305-9d25-2742006cffd6 2020-03-11T22:36:26.752 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:26.748+0000 7f6790198ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:26.755 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:26.751+0000 7f6790198ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:26.757 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:26.753+0000 7f6790198ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:26.758 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:26.754+0000 7f6790198ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:26.758 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:26.754+0000 7f6790198ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:26.759 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:26.754+0000 7f6790198ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:36:26.821 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:26.817+0000 7f6790198ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:36:29.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:36:29.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:36:29.765 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:36:29.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:36:29.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:36:30.224 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:36:30.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:36:30.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:30.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:36:30.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:36:30.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:36:30.251 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:30.247+0000 7f10ae7e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:30.252 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:30.249+0000 7f10ae7e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:30.253 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:30.250+0000 7f10ae7e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:30.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:36:30.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:30.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:36:30.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:30.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:30.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:30.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:30.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:30.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:30.572 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:36:30.769 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:30.765+0000 7f10ae7e1ec0 -1 Falling back to public interface 2020-03-11T22:36:30.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:31.558 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:36:31.554+0000 7f10ae7e1ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:36:31.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:31.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:31.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:31.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:31.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:31.925 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:36:32.270 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26215,v1:127.0.0.1:6819/26215] [v2:127.0.0.1:6820/26215,v1:127.0.0.1:6821/26215] exists,up ad75aea1-426e-4305-9d25-2742006cffd6 2020-03-11T22:36:32.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:32.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:32.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:32.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:36:32.967 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2020-03-11T22:36:32.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:36:33.568 INFO:tasks.workunit.client.0.smithi130.stderr:noin is set 2020-03-11T22:36:33.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:36:34.136 INFO:tasks.workunit.client.0.smithi130.stderr:nodown is set 2020-03-11T22:36:34.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:36:34.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:36:34.652 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2020-03-11T22:36:34.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:36:35.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:36:36.373 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 3 2020-03-11T22:36:36.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:36:36.990 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 min_size to 2 2020-03-11T22:36:36.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:36:36.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:36.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:37.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:37.344 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:36:37.344 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:36:37.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:37.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:37.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:37.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:36:37.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:36:37.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:37.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:37.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:36:37.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:36:37.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:37.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:37.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:36:37.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509444' 2020-03-11T22:36:37.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:37.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:36:37.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:37.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:37.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:36:37.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:37.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:36:37.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:36:37.736 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:36:37.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:38.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:36:38.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:39.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:39.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:39.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:36:39.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:39.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:36:39.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:39.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:39.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:36:39.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:39.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:36:39.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:36:39.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:39.456 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:36:39.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:36:39.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:39.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:36:39.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:39.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:39.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:36:39.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:39.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:36:39.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:36:39.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:39.800 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:36:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509445 -lt 64424509444 2020-03-11T22:36:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:40.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:36:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:40.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:40.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:40.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:40.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:40.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:40.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:40.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:40.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:40.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:40.579 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:36:40.579 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:36:40.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:40.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:40.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:40.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:36:40.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:36:40.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:40.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:40.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:36:40.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:36:40.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:40.838 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:40.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509447 2020-03-11T22:36:40.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2020-03-11T22:36:40.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:40.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:36:40.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:40.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:40.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:36:40.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:40.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:36:40.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:36:40.970 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:36:40.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:41.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836489 2020-03-11T22:36:41.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:42.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:42.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:42.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:36:42.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:42.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:36:42.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:42.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:42.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:36:42.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:42.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:36:42.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:36:42.665 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:36:42.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:43.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:36:43.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:43.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509447 2020-03-11T22:36:43.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:43.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:43.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509447 2020-03-11T22:36:43.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:43.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509447 2020-03-11T22:36:43.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2020-03-11T22:36:43.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:43.011 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509447 2020-03-11T22:36:43.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509448 -lt 64424509447 2020-03-11T22:36:43.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:43.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:43.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:44.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:36:44.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:44.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:44.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:44.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:36:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:36:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:36:44.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:36:44.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:36:44.908 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:36:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:36:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:36:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:44.917 INFO:tasks.workunit.client.0.smithi130.stdout:primary and soon to be divergent is 1 2020-03-11T22:36:45.235 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:45.235 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:36:35.634360+0000 0'0 24:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:36:45.235 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:36:45.235 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:45.236 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:36:45.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:36:45.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:45.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:36:45.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:36:45.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:45.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:36:45.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:36:45.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:36:45.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:36:45.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:36:45.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:36:45.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:36:45.247 INFO:tasks.workunit.client.0.smithi130.stdout:writing initial objects 2020-03-11T22:36:45.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:36:45.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:36:45.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:36:45.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:36:45.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:36:45.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:36:45.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:36:45.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:36:45.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:36:45.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:36:45.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:36:45.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:36:45.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:36:45.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:36:45.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:45.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:36:46.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:36:46.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:36:46.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:36:46.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:36:46.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:36:46.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:36:46.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:36:46.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:36:46.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:36:46.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:36:46.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:36:46.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:36:46.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:36:46.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:36:46.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:36:46.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:36:46.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:36:46.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:36:46.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:46.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:36:47.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:36:47.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:36:47.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:36:47.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:36:47.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:36:47.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:36:47.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:36:47.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:36:47.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:36:47.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:36:47.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:36:47.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:36:47.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:47.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:36:48.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:36:48.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:36:48.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:36:48.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:36:48.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:36:48.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:36:48.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:36:48.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:36:48.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:36:48.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:36:48.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:36:48.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:36:48.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:36:48.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:36:48.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:36:48.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:36:48.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:36:48.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:48.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:36:49.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:36:49.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:36:49.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:36:49.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:36:49.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:36:49.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:36:49.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:36:49.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:36:49.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:36:49.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:36:49.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:36:49.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:36:49.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:36:49.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:36:49.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:36:49.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:36:49.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:49.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:36:50.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:36:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:36:50.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:36:50.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:36:50.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:36:50.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:36:50.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:36:50.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:36:50.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:36:50.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:36:50.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:36:50.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:36:50.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:36:50.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:36:50.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:36:50.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:36:50.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:36:50.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:50.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:50.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:50.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:50.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:50.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:36:50.823 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:50.823 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:50.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:50.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:50.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:50.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:50.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:50.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:50.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:50.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:51.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:51.219 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:36:51.220 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:36:51.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:51.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:51.220 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:51.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:36:51.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:36:51.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:51.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:51.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:36:51.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:36:51.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:51.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:51.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:36:51.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:36:51.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:51.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:36:51.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:51.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:51.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:36:51.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:51.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:36:51.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:36:51.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:51.610 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:36:51.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:36:51.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:52.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:52.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:53.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:36:53.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:54.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:36:54.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:54.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:36:54.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:54.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:54.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:54.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:54.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:36:54.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:36:54.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:36:54.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:54.687 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:36:55.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:36:55.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:55.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:55.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:55.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:55.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:36:55.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:55.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:36:55.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:36:55.033 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:36:55.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:55.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509452 -lt 64424509451 2020-03-11T22:36:55.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:55.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:55.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:55.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:55.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:55.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:55.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:55.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:55.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:55.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:55.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:56.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:36:56.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:36:56.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:36:56.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:36:56.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:36:56.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:36:56.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:56.958 INFO:tasks.workunit.client.0.smithi130.stdout:blackholing osds 0 2 2020-03-11T22:36:57.277 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:57.277 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:36:35.634360+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:36:57.277 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:36:57.277 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:57.278 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:36:57.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:57.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:36:57.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:36:57.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:36:57.288 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:57.288 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:57.288 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:57.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:36:57.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:36:57.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:36:57.375 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:36:57.375 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:57.376 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:36:57.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:57.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:36:57.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:36:57.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:36:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:36:57.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:36:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:36:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:36:57.472 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:36:57.472 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:36:57.472 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:36:57.478 INFO:tasks.workunit.client.0.smithi130.stdout:create new divergent objects 2020-03-11T22:36:57.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:36:57.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:36:57.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:36:57.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:36:57.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:36:57.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:36:57.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:36:57.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:36:57.482 INFO:tasks.workunit.client.0.smithi130.stdout:writing divergent objects 2020-03-11T22:36:57.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:36:57.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:36:57.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:36:57.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:36:57.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:36:57.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:36:57.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:36:57.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:36:57.484 INFO:tasks.workunit.client.0.smithi130.stdout:remove divergent objects 2020-03-11T22:36:57.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:36:57.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:36:57.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:36:57.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:36:57.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:36:57.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:36:57.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:37:07.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:37:07.489 INFO:tasks.workunit.client.0.smithi130.stderr:rados: no process found 2020-03-11T22:37:07.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:37:07.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:07.490 INFO:tasks.workunit.client.0.smithi130.stdout:killing all the osds 2020-03-11T22:37:07.818 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:07.818 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:36:35.634360+0000 28'106 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:37:07.818 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:37:07.819 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:07.820 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:37:07.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:37:07.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:07.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:07.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:07.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:07.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:37:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:37:08.421 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2020-03-11T22:37:08.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:37:08.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:37:08.904 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already down. 2020-03-11T22:37:08.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:37:08.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:37:09.421 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already down. 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:37:09.928 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2020-03-11T22:37:09.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:09.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:37:10.448 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2020-03-11T22:37:10.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:37:10.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:10.456 INFO:tasks.workunit.client.0.smithi130.stdout:bringing up non_divergent 0 2 2020-03-11T22:37:10.775 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:10.775 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:36:35.634360+0000 28'106 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:37:10.775 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:37:10.775 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:37:10.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:10.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:37:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:37:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:37:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:10.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:10.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:10.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:10.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:10.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:10.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:37:10.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:37:10.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:37:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:10.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:10.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:10.794 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:37:10.814 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:10.811+0000 7f0dec127ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:10.815 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:10.813+0000 7f0dec127ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:10.816 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:10.814+0000 7f0dec127ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:11.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:37:11.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:11.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:11.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:11.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:11.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:11.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:11.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:11.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:11.135 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:37:11.331 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:11.328+0000 7f0dec127ec0 -1 Falling back to public interface 2020-03-11T22:37:11.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:12.178 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:12.176+0000 7f0dec127ec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:37:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:12.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:12.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:12.479 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:37:12.835 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30293,v1:127.0.0.1:6803/30293] [v2:127.0.0.1:6804/30293,v1:127.0.0.1:6805/30293] exists,up 1b8f82d0-d42b-4d93-ab95-f22ad2bc035e 2020-03-11T22:37:12.836 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:37:12.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:12.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:12.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:12.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:12.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:37:12.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:12.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:12.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:37:12.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:12.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:37:12.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:12.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:12.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:12.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:12.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:37:12.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:37:12.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:12.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:12.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:12.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:12.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:12.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:12.842 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:12.842 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:12.842 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:37:12.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:37:12.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:37:12.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:12.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:12.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:12.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:12.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:12.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:12.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:12.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:12.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:37:12.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:37:12.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:12.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:37:12.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:37:12.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:12.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:12.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:12.857 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:12.855+0000 7f5c0bdb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:12.859 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:12.856+0000 7f5c0bdb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:12.859 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:12.857+0000 7f5c0bdb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:13.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:37:13.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:13.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:13.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:13.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:13.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:13.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:13.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:13.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:13.219 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:37:13.375 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:13.372+0000 7f5c0bdb7ec0 -1 Falling back to public interface 2020-03-11T22:37:13.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:14.269 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:37:14.267+0000 7f5c0bdb7ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:37:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:14.584 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:37:14.932 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30487,v1:127.0.0.1:6811/30487] [v2:127.0.0.1:6812/30487,v1:127.0.0.1:6813/30487] exists,up ad75aea1-426e-4305-9d25-2742006cffd6 2020-03-11T22:37:14.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:14.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:14.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:14.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:14.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:37:15.437 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already in. 2020-03-11T22:37:15.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:15.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:37:15.889 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2020-03-11T22:37:15.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:15.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:37:15.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:15.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:15.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:15.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:15.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:15.900 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:15.900 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:15.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:15.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:15.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:15.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:15.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:15.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:15.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:16.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:16.302 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:37:16.302 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:37:16.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:16.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:16.302 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:16.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:37:16.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:37:16.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:16.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:16.534 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:37:16.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:37:16.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:37:16.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:16.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:16.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:37:16.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:37:16.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:16.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:37:16.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:16.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:16.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:37:16.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:16.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:37:16.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:37:16.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:16.675 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:37:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:37:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:18.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:18.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:18.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:37:18.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:18.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:37:18.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:18.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:18.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:37:18.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:18.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:37:18.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:37:18.367 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 2020-03-11T22:37:18.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:18.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 2020-03-11T22:37:18.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672976: unary operator expected 2020-03-11T22:37:18.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:18.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:37:18.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:18.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:18.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:37:18.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:18.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:37:18.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:37:18.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:18.859 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:37:19.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:37:19.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:19.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:19.205 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:19.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:19.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:19.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:19.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:19.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:19.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:19.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:19.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:19.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:19.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:19.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:19.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:20.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:20.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:37:20.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:37:20.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:37:20.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:20.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:37:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:20.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:20.861 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:20.861 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:20.861 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:21.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:21.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:21.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:21.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:21.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:21.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:21.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:21.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:21.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:21.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:21.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:21.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:37:21.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:21.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:37:21.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:21.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:21.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:21.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:21.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:21.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:21.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:21.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:22.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:22.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:22.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:22.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:22.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:22.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:22.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:22.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:22.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:22.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:22.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:22.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:23.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:23.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:23.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:37:23.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:23.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:37:23.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:23.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:23.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:23.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:23.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:23.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:23.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:23.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:23.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:23.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:23.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:23.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:24.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:24.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:24.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:24.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:24.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:24.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:24.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:24.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:24.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:24.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:24.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:24.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:37:24.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:24.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:37:25.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:25.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:25.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:25.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:25.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:25.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:25.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:25.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:25.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:25.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:25.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:25.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:26.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:26.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:26.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:26.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:26.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:26.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:26.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:26.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:26.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:26.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:26.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:26.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:37:26.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:26.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:37:28.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:28.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:28.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:28.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:28.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:28.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:28.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:28.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:28.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:28.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:28.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:28.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:29.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:29.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:29.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:29.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:29.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:29.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:29.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:29.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:29.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:29.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:29.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:29.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:37:29.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:29.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:37:32.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:32.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:32.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:32.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:32.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:32.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:32.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:32.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:33.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:33.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:33.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:33.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:33.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:33.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:33.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:33.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:33.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:33.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:33.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:33.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:33.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:34.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:34.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:34.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:37:34.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:34.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:37:40.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:40.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:40.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:40.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:40.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:40.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:40.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:40.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:40.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:40.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:40.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:40.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:41.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:41.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:41.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:41.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:41.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:41.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:41.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:41.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:41.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:41.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:41.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:41.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:37:41.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:41.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:37:49.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:49.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:49.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:49.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:49.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:49.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:49.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:49.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:49.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:49.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:49.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:49.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:49.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:49.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:49.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:49.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:49.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:49.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:49.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:49.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:49.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:50.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:50.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:50.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:37:50.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:37:50.673 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:37:50.674 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_fingerprint": "d9851a65-030e-401a-b359-f38a9a034f2f", 2020-03-11T22:37:50.674 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:37:50.674 INFO:tasks.workunit.client.0.smithi130.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:37:50.674 INFO:tasks.workunit.client.0.smithi130.stdout: "timestamp": "2020-03-11T22:37:50.670561+0000", 2020-03-11T22:37:50.674 INFO:tasks.workunit.client.0.smithi130.stdout: "tag": "", 2020-03-11T22:37:50.674 INFO:tasks.workunit.client.0.smithi130.stdout: "health": { 2020-03-11T22:37:50.674 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "HEALTH_WARN", 2020-03-11T22:37:50.675 INFO:tasks.workunit.client.0.smithi130.stdout: "checks": { 2020-03-11T22:37:50.675 INFO:tasks.workunit.client.0.smithi130.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:37:50.675 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:50.675 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:37:50.675 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:37:50.675 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 17 2020-03-11T22:37:50.675 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [], 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: "OSD_DOWN": { 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 osds down", 2020-03-11T22:37:50.676 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:37:50.677 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.677 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:37:50.677 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.677 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "osd.1 (root=default,host=smithi130) is down" 2020-03-11T22:37:50.677 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.677 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.678 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:37:50.678 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.678 INFO:tasks.workunit.client.0.smithi130.stdout: "PG_DEGRADED": { 2020-03-11T22:37:50.678 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:50.678 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:37:50.678 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:37:50.678 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:37:50.679 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.679 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:37:50.679 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.679 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:37:50.679 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.679 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.679 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:37:50.680 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.680 INFO:tasks.workunit.client.0.smithi130.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:37:50.680 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:50.680 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:37:50.680 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:37:50.680 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:37:50.680 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.681 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.682 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:37:50.682 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.682 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.682 INFO:tasks.workunit.client.0.smithi130.stdout: "mutes": [] 2020-03-11T22:37:50.682 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.682 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_first_committed": 1, 2020-03-11T22:37:50.682 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_last_committed": 1, 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap": { 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:36:12.511238Z", 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:36:12.511238Z", 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release": 15, 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:37:50.683 INFO:tasks.workunit.client.0.smithi130.stdout: "features": { 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: "persistent": [ 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: "kraken", 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: "luminous", 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: "mimic", 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap-prune", 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: "nautilus", 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: "octopus" 2020-03-11T22:37:50.684 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.685 INFO:tasks.workunit.client.0.smithi130.stdout: "optional": [] 2020-03-11T22:37:50.685 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.685 INFO:tasks.workunit.client.0.smithi130.stdout: "mons": [ 2020-03-11T22:37:50.685 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.685 INFO:tasks.workunit.client.0.smithi130.stdout: "rank": 0, 2020-03-11T22:37:50.685 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "a", 2020-03-11T22:37:50.685 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:37:50.686 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.686 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.686 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.686 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:37:50.686 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 0 2020-03-11T22:37:50.686 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.686 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.687 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.687 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:37:50.687 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:37:50.687 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 0, 2020-03-11T22:37:50.688 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 0 2020-03-11T22:37:50.688 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.688 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.688 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.688 INFO:tasks.workunit.client.0.smithi130.stdout: "quorum": [ 2020-03-11T22:37:50.689 INFO:tasks.workunit.client.0.smithi130.stdout: 0 2020-03-11T22:37:50.689 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.689 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap": { 2020-03-11T22:37:50.689 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 39, 2020-03-11T22:37:50.689 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:37:50.689 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:36:13.224040+0000", 2020-03-11T22:37:50.690 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:37:15.925422+0000", 2020-03-11T22:37:50.690 INFO:tasks.workunit.client.0.smithi130.stdout: "last_up_change": "2020-03-11T22:37:14.287519+0000", 2020-03-11T22:37:50.690 INFO:tasks.workunit.client.0.smithi130.stdout: "last_in_change": "2020-03-11T22:37:15.698163+0000", 2020-03-11T22:37:50.690 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:37:50.690 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_num": 5800384, 2020-03-11T22:37:50.690 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_set": [ 2020-03-11T22:37:50.690 INFO:tasks.workunit.client.0.smithi130.stdout: "nodown", 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: "noin", 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: "noout", 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: "pglog_hardlimit", 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snapdirs", 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_deletes", 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: "sortbitwise" 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.691 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_version": 7, 2020-03-11T22:37:50.692 INFO:tasks.workunit.client.0.smithi130.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:37:50.692 INFO:tasks.workunit.client.0.smithi130.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:37:50.692 INFO:tasks.workunit.client.0.smithi130.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:37:50.692 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_snapshot": "", 2020-03-11T22:37:50.692 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_max": 1, 2020-03-11T22:37:50.692 INFO:tasks.workunit.client.0.smithi130.stdout: "max_osd": 3, 2020-03-11T22:37:50.692 INFO:tasks.workunit.client.0.smithi130.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: "min_compat_client": "jewel", 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: "require_osd_release": "octopus", 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: "pools": [ 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_name": "test", 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: "create_time": "2020-03-11T22:36:34.399712+0000", 2020-03-11T22:37:50.693 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": 1, 2020-03-11T22:37:50.694 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_names": "hashpspool", 2020-03-11T22:37:50.694 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:37:50.694 INFO:tasks.workunit.client.0.smithi130.stdout: "size": 3, 2020-03-11T22:37:50.694 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 2, 2020-03-11T22:37:50.694 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_rule": 0, 2020-03-11T22:37:50.694 INFO:tasks.workunit.client.0.smithi130.stdout: "object_hash": 2, 2020-03-11T22:37:50.694 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num": 1, 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num": 1, 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num_target": 1, 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_target": 1, 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_pending": 1, 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pg_merge_meta": { 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "source_pgid": "0.0", 2020-03-11T22:37:50.695 INFO:tasks.workunit.client.0.smithi130.stdout: "ready_epoch": 0, 2020-03-11T22:37:50.696 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 0, 2020-03-11T22:37:50.696 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 0, 2020-03-11T22:37:50.696 INFO:tasks.workunit.client.0.smithi130.stdout: "source_version": "0'0", 2020-03-11T22:37:50.696 INFO:tasks.workunit.client.0.smithi130.stdout: "target_version": "0'0" 2020-03-11T22:37:50.696 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.696 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "28", 2020-03-11T22:37:50.696 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend": "0", 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "auid": 0, 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_seq": 0, 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_epoch": 0, 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_snaps": [], 2020-03-11T22:37:50.697 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps": "[]", 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_bytes": 0, 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_objects": 0, 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "tiers": [], 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "tier_of": -1, 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "read_tier": -1, 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "write_tier": -1, 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_mode": "none", 2020-03-11T22:37:50.698 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_bytes": 0, 2020-03-11T22:37:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_objects": 0, 2020-03-11T22:37:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:37:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:37:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:37:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_flush_age": 0, 2020-03-11T22:37:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_evict_age": 0, 2020-03-11T22:37:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profile": "", 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_params": { 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "none" 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_period": 0, 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_count": 0, 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: "use_gmt_hitset": true, 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:37:50.700 INFO:tasks.workunit.client.0.smithi130.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:37:50.701 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:37:50.701 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:37:50.701 INFO:tasks.workunit.client.0.smithi130.stdout: "grade_table": [], 2020-03-11T22:37:50.701 INFO:tasks.workunit.client.0.smithi130.stdout: "stripe_width": 0, 2020-03-11T22:37:50.701 INFO:tasks.workunit.client.0.smithi130.stdout: "expected_num_objects": 0, 2020-03-11T22:37:50.701 INFO:tasks.workunit.client.0.smithi130.stdout: "fast_read": false, 2020-03-11T22:37:50.701 INFO:tasks.workunit.client.0.smithi130.stdout: "options": {}, 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: "application_metadata": {} 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: "osds": [ 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "1b8f82d0-d42b-4d93-ab95-f22ad2bc035e", 2020-03-11T22:37:50.702 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:37:50.703 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:37:50.703 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:37:50.703 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:37:50.703 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 5, 2020-03-11T22:37:50.703 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 28, 2020-03-11T22:37:50.703 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 34, 2020-03-11T22:37:50.703 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 38, 2020-03-11T22:37:50.704 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 29, 2020-03-11T22:37:50.704 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:37:50.704 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:37:50.704 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.704 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.704 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.704 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:37:50.705 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.705 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.705 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.705 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.705 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:37:50.705 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.705 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.706 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.706 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.706 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:37:50.706 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.706 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.706 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.706 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:37:50.707 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.707 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.707 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.707 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.707 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:37:50.707 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.708 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:37:50.709 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.709 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.709 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.709 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.709 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:37:50.709 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.709 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:37:50.710 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.711 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.711 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.711 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.711 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:37:50.711 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:37:50.711 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.711 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6803/30293", 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6805/30293", 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30293", 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30293", 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:37:50.712 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:37:50.713 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.713 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.713 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.713 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:37:50.713 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "e938a584-5c23-4b9f-8a94-a38336bc98c0", 2020-03-11T22:37:50.713 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 0, 2020-03-11T22:37:50.713 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 0, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 0, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 10, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 22, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 30, 2020-03-11T22:37:50.714 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.715 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.716 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.716 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:50.716 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.716 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.716 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.716 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.716 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.717 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.718 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:50.718 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.718 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.718 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.718 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.718 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:50.718 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.719 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.719 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.719 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:50.719 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.719 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.719 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.719 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.720 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:50.720 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.720 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.720 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.720 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.720 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.721 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.722 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:50.722 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:37:50.722 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.722 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.722 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.722 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/25860", 2020-03-11T22:37:50.723 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/25860", 2020-03-11T22:37:50.723 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25860", 2020-03-11T22:37:50.723 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25860", 2020-03-11T22:37:50.723 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:37:50.723 INFO:tasks.workunit.client.0.smithi130.stdout: "exists" 2020-03-11T22:37:50.723 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.723 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.724 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.724 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:37:50.724 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "ad75aea1-426e-4305-9d25-2742006cffd6", 2020-03-11T22:37:50.724 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:37:50.724 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:37:50.724 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:37:50.725 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:37:50.725 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 15, 2020-03-11T22:37:50.725 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 28, 2020-03-11T22:37:50.725 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 35, 2020-03-11T22:37:50.726 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 0, 2020-03-11T22:37:50.726 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 31, 2020-03-11T22:37:50.726 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:37:50.726 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:37:50.726 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.726 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.726 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.727 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.728 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.728 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.728 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:37:50.728 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.728 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.728 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.728 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.729 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.730 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.730 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:50.730 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.730 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.730 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.730 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:50.730 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.732 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.733 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:37:50.733 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:50.733 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:37:50.733 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.733 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.733 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.733 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/30487", 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/30487", 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30487", 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30487", 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.734 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.735 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.735 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_xinfo": [ 2020-03-11T22:37:50.735 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.735 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:37:50.735 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:37:08.235832+0000", 2020-03-11T22:37:50.735 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:37:50.735 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:20.177768+0000", 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:37:50.736 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:37:08.733445+0000", 2020-03-11T22:37:50.737 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:37:50.737 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:37:50.737 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:37:50.737 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:37:50.737 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:26.520312+0000", 2020-03-11T22:37:50.737 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:37:50.737 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.738 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.738 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:37:50.738 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:37:09.228952+0000", 2020-03-11T22:37:50.738 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:37:50.738 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:37:50.738 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:37:50.739 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:37:50.739 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:32.531833+0000", 2020-03-11T22:37:50.739 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:37:50.739 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.739 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.739 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap": [], 2020-03-11T22:37:50.739 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap_items": [], 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_temp": [], 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_temp": [], 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "blacklist": {}, 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profiles": { 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "default": { 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "2", 2020-03-11T22:37:50.740 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:37:50.741 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van" 2020-03-11T22:37:50.741 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.741 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.741 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps_queue": [], 2020-03-11T22:37:50.741 INFO:tasks.workunit.client.0.smithi130.stdout: "new_removed_snaps": [], 2020-03-11T22:37:50.741 INFO:tasks.workunit.client.0.smithi130.stdout: "new_purged_snaps": [], 2020-03-11T22:37:50.741 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_node_flags": {}, 2020-03-11T22:37:50.742 INFO:tasks.workunit.client.0.smithi130.stdout: "device_class_flags": {} 2020-03-11T22:37:50.742 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.742 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_metadata": [ 2020-03-11T22:37:50.742 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.742 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:37:50.742 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:37:50.743 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6804/30293,v1:127.0.0.1:6805/30293]", 2020-03-11T22:37:50.743 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:37:50.743 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:37:50.743 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:50.743 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:50.743 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:50.743 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:50.744 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:50.745 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:37:50.745 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:50.745 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:50.745 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:37:50.745 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:37:50.745 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:37:50.745 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6802/30293,v1:127.0.0.1:6803/30293]", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30293,v1:127.0.0.1:6809/30293]", 2020-03-11T22:37:50.746 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30293,v1:127.0.0.1:6807/30293]", 2020-03-11T22:37:50.747 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:37:50.747 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:37:50.747 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:50.747 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:50.747 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:50.747 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:37:50.747 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.748 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:37:50.749 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:37:50.749 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/25860,v1:127.0.0.1:6813/25860]", 2020-03-11T22:37:50.749 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:37:50.749 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:37:50.749 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:50.749 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:50.750 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:50.750 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:50.750 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:50.750 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:50.750 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:50.750 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:37:50.750 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:50.751 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:50.751 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:50.751 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:50.751 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:37:50.751 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:50.751 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:50.751 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:37:50.752 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:37:50.752 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:37:50.752 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:50.752 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:37:50.752 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:37:50.752 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:50.753 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:37:50.753 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/25860,v1:127.0.0.1:6811/25860]", 2020-03-11T22:37:50.753 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:37:50.753 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25860,v1:127.0.0.1:6817/25860]", 2020-03-11T22:37:50.753 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25860,v1:127.0.0.1:6815/25860]", 2020-03-11T22:37:50.753 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:37:50.753 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:37:50.754 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:50.754 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:50.754 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:50.754 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:37:50.754 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:50.754 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:37:50.755 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:37:50.756 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/30487,v1:127.0.0.1:6813/30487]", 2020-03-11T22:37:50.756 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:37:50.756 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:37:50.756 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:50.756 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:50.757 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:50.757 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:50.757 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:50.757 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:50.757 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:50.757 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:37:50.757 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:37:50.758 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/30487,v1:127.0.0.1:6811/30487]", 2020-03-11T22:37:50.759 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:37:50.760 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30487,v1:127.0.0.1:6817/30487]", 2020-03-11T22:37:50.760 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30487,v1:127.0.0.1:6815/30487]", 2020-03-11T22:37:50.760 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:37:50.760 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:37:50.760 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:37:50.760 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:37:50.760 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:37:50.761 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.762 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.762 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_first_committed": 1, 2020-03-11T22:37:50.762 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_last_committed": 39, 2020-03-11T22:37:50.762 INFO:tasks.workunit.client.0.smithi130.stdout: "crushmap": { 2020-03-11T22:37:50.762 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": [ 2020-03-11T22:37:50.762 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.762 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.0", 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.1", 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:37:50.763 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.764 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.764 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:37:50.764 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.2", 2020-03-11T22:37:50.764 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:37:50.764 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.764 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.764 INFO:tasks.workunit.client.0.smithi130.stdout: "types": [ 2020-03-11T22:37:50.765 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.765 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 0, 2020-03-11T22:37:50.765 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd" 2020-03-11T22:37:50.765 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.765 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.765 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:37:50.765 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "host" 2020-03-11T22:37:50.766 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.766 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 2, 2020-03-11T22:37:50.766 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "chassis" 2020-03-11T22:37:50.766 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.766 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 3, 2020-03-11T22:37:50.767 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "rack" 2020-03-11T22:37:50.767 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.767 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.767 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 4, 2020-03-11T22:37:50.767 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "row" 2020-03-11T22:37:50.767 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.767 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 5, 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pdu" 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 6, 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pod" 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.768 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 7, 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "room" 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 8, 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "datacenter" 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.770 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 9, 2020-03-11T22:37:50.770 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "zone" 2020-03-11T22:37:50.770 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.770 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.770 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 10, 2020-03-11T22:37:50.770 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "region" 2020-03-11T22:37:50.770 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.771 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.771 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:37:50.771 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "root" 2020-03-11T22:37:50.771 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.771 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.771 INFO:tasks.workunit.client.0.smithi130.stdout: "buckets": [ 2020-03-11T22:37:50.772 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.772 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -1, 2020-03-11T22:37:50.773 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default", 2020-03-11T22:37:50.773 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:37:50.773 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:37:50.773 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:37:50.774 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:37:50.774 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:37:50.774 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:37:50.774 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.775 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:37:50.775 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:37:50.775 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:37:50.775 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.775 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.776 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.776 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.776 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -2, 2020-03-11T22:37:50.776 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default~hdd", 2020-03-11T22:37:50.776 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:37:50.777 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:37:50.777 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:37:50.777 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:37:50.777 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:37:50.778 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:37:50.778 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.778 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:37:50.778 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:37:50.778 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:37:50.779 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.779 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.779 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.779 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.779 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:37:50.780 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130", 2020-03-11T22:37:50.780 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:37:50.780 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:37:50.780 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:37:50.781 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:37:50.781 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:37:50.781 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:37:50.781 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.781 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:37:50.782 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:37:50.782 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:37:50.782 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.782 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.782 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:37:50.783 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:37:50.783 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:37:50.783 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.783 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.784 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:37:50.784 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:37:50.784 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:37:50.784 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.784 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.785 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.785 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.785 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:37:50.785 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130~hdd", 2020-03-11T22:37:50.786 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:37:50.786 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:37:50.786 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:37:50.786 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:37:50.786 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:37:50.787 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:37:50.787 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.787 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:37:50.787 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:37:50.788 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:37:50.788 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.788 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.788 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:37:50.788 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:37:50.789 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:37:50.789 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.789 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.789 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:37:50.790 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:37:50.790 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:37:50.790 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.790 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.790 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.791 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.791 INFO:tasks.workunit.client.0.smithi130.stdout: "rules": [ 2020-03-11T22:37:50.791 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.791 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 0, 2020-03-11T22:37:50.792 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "replicated_rule", 2020-03-11T22:37:50.792 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 0, 2020-03-11T22:37:50.792 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:37:50.792 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 1, 2020-03-11T22:37:50.792 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 10, 2020-03-11T22:37:50.793 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:37:50.793 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.793 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:37:50.793 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:37:50.793 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:37:50.794 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.794 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.794 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_firstn", 2020-03-11T22:37:50.794 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:37:50.795 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:37:50.795 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.795 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.795 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:37:50.795 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.796 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:37:50.796 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.796 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.796 INFO:tasks.workunit.client.0.smithi130.stdout: "tunables": { 2020-03-11T22:37:50.796 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_tries": 0, 2020-03-11T22:37:50.796 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:37:50.797 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_total_tries": 50, 2020-03-11T22:37:50.797 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:37:50.797 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:37:50.797 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_stable": 1, 2020-03-11T22:37:50.798 INFO:tasks.workunit.client.0.smithi130.stdout: "straw_calc_version": 1, 2020-03-11T22:37:50.798 INFO:tasks.workunit.client.0.smithi130.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:37:50.798 INFO:tasks.workunit.client.0.smithi130.stdout: "profile": "jewel", 2020-03-11T22:37:50.798 INFO:tasks.workunit.client.0.smithi130.stdout: "optimal_tunables": 1, 2020-03-11T22:37:50.798 INFO:tasks.workunit.client.0.smithi130.stdout: "legacy_tunables": 0, 2020-03-11T22:37:50.799 INFO:tasks.workunit.client.0.smithi130.stdout: "minimum_required_version": "jewel", 2020-03-11T22:37:50.799 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables": 1, 2020-03-11T22:37:50.799 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables2": 1, 2020-03-11T22:37:50.800 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v2_rules": 0, 2020-03-11T22:37:50.800 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables3": 1, 2020-03-11T22:37:50.800 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v3_rules": 0, 2020-03-11T22:37:50.800 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v4_buckets": 1, 2020-03-11T22:37:50.800 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables5": 1, 2020-03-11T22:37:50.801 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v5_rules": 0 2020-03-11T22:37:50.801 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.801 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_args": {} 2020-03-11T22:37:50.801 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.802 INFO:tasks.workunit.client.0.smithi130.stdout: "fsmap": { 2020-03-11T22:37:50.802 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:37:50.802 INFO:tasks.workunit.client.0.smithi130.stdout: "default_fscid": -1, 2020-03-11T22:37:50.803 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": { 2020-03-11T22:37:50.803 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": {}, 2020-03-11T22:37:50.803 INFO:tasks.workunit.client.0.smithi130.stdout: "ro_compat": {}, 2020-03-11T22:37:50.803 INFO:tasks.workunit.client.0.smithi130.stdout: "incompat": { 2020-03-11T22:37:50.803 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_1": "base v0.20", 2020-03-11T22:37:50.804 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:37:50.804 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:37:50.804 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:37:50.804 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:37:50.805 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:37:50.805 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_8": "no anchor table", 2020-03-11T22:37:50.805 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_9": "file layout v2", 2020-03-11T22:37:50.805 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:37:50.805 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.806 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.806 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_flags": { 2020-03-11T22:37:50.806 INFO:tasks.workunit.client.0.smithi130.stdout: "enable_multiple": false, 2020-03-11T22:37:50.806 INFO:tasks.workunit.client.0.smithi130.stdout: "ever_enabled_multiple": false 2020-03-11T22:37:50.806 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.807 INFO:tasks.workunit.client.0.smithi130.stdout: "standbys": [], 2020-03-11T22:37:50.807 INFO:tasks.workunit.client.0.smithi130.stdout: "filesystems": [] 2020-03-11T22:37:50.807 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.807 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:37:50.807 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:37:50.807 INFO:tasks.workunit.client.0.smithi130.stdout: "auth": { 2020-03-11T22:37:50.808 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:37:50.808 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 5, 2020-03-11T22:37:50.808 INFO:tasks.workunit.client.0.smithi130.stdout: "num_secrets": 3 2020-03-11T22:37:50.808 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.809 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:37:50.809 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_active": 1, 2020-03-11T22:37:50.809 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_unknown": 0, 2020-03-11T22:37:50.809 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osd": 3, 2020-03-11T22:37:50.810 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_sum": { 2020-03-11T22:37:50.810 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:37:50.810 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48300, 2020-03-11T22:37:50.810 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:37:50.810 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:37:50.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:37:50.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:50.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:37:50.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:50.811 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:50.812 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:50.812 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:50.812 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:37:50.812 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:37:50.813 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:37:50.813 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 100, 2020-03-11T22:37:50.813 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 100, 2020-03-11T22:37:50.813 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:50.813 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:50.814 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:50.814 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:50.814 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:50.814 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:50.815 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:37:50.815 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:50.815 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:50.815 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:37:50.815 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:37:50.816 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:37:50.816 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:37:50.816 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:37:50.816 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:50.816 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:50.817 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:50.817 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:50.817 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:50.817 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:50.818 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:50.818 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:50.818 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:50.818 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:37:50.818 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:37:50.819 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.819 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:37:50.819 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:37:50.819 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:37:50.819 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:37:50.820 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 0, 2020-03-11T22:37:50.820 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 0, 2020-03-11T22:37:50.820 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:37:50.820 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:50.821 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:37:50.821 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:37:50.821 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:37:50.822 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.822 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 16, 2020-03-11T22:37:50.822 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 16, 2020-03-11T22:37:50.822 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 2, 2020-03-11T22:37:50.822 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 2, 2020-03-11T22:37:50.823 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 0 2020-03-11T22:37:50.823 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.823 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum": { 2020-03-11T22:37:50.823 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:37:50.824 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:37:50.824 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:37:50.824 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:37:50.824 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:50.824 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:50.825 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:37:50.825 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3179072, 2020-03-11T22:37:50.825 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 33024, 2020-03-11T22:37:50.825 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 13, 2020-03-11T22:37:50.826 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145714, 2020-03-11T22:37:50.826 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393728, 2020-03-11T22:37:50.826 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:37:50.826 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:37:50.826 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318867177472, 2020-03-11T22:37:50.827 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:50.827 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 33816576, 2020-03-11T22:37:50.827 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 402091, 2020-03-11T22:37:50.827 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:37:50.827 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:50.828 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:37:50.828 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 14090, 2020-03-11T22:37:50.828 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221211382 2020-03-11T22:37:50.828 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.828 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:37:50.829 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:50.829 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:50.829 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:50.829 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:37:50.830 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:37:50.830 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:37:50.830 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.830 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:37:50.830 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:50.831 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:50.831 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:50.831 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:37:50.831 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.832 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:37:50.832 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:37:50.832 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.832 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum_by_class": { 2020-03-11T22:37:50.832 INFO:tasks.workunit.client.0.smithi130.stdout: "hdd": { 2020-03-11T22:37:50.833 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:37:50.833 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:37:50.833 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:37:50.833 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:37:50.834 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:50.834 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:50.834 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:37:50.834 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3179072, 2020-03-11T22:37:50.834 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 33024, 2020-03-11T22:37:50.835 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 13, 2020-03-11T22:37:50.835 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145714, 2020-03-11T22:37:50.835 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393728, 2020-03-11T22:37:50.835 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:37:50.835 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:37:50.836 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318867177472, 2020-03-11T22:37:50.836 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:50.836 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 33816576, 2020-03-11T22:37:50.836 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 402091, 2020-03-11T22:37:50.837 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:37:50.837 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:50.837 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:37:50.837 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 14090, 2020-03-11T22:37:50.837 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221211382 2020-03-11T22:37:50.838 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.838 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:37:50.838 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:50.838 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:50.838 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:50.839 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:37:50.839 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:37:50.839 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:37:50.840 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.840 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:37:50.840 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:50.840 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:50.840 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:50.841 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:37:50.841 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.841 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:37:50.841 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:37:50.842 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.842 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.842 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_stats": [ 2020-03-11T22:37:50.842 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.842 INFO:tasks.workunit.client.0.smithi130.stdout: "poolid": 1, 2020-03-11T22:37:50.843 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:37:50.843 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:37:50.843 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48300, 2020-03-11T22:37:50.843 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:37:50.843 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:37:50.844 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:37:50.844 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:50.844 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:37:50.844 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:50.845 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:50.845 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:50.845 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:50.845 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:37:50.845 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:37:50.846 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:37:50.846 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 100, 2020-03-11T22:37:50.846 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 100, 2020-03-11T22:37:50.846 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:50.846 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:50.847 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:50.847 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:50.847 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:50.847 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:50.847 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:37:50.848 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:50.848 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:50.848 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:37:50.848 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:37:50.849 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:37:50.849 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:37:50.849 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:37:50.849 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:50.850 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:50.850 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:50.850 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:50.850 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:50.850 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:50.850 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:50.850 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:50.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:50.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:37:50.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:37:50.851 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.851 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:37:50.851 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:37:50.851 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:37:50.852 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:37:50.852 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 19529728, 2020-03-11T22:37:50.852 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 143228, 2020-03-11T22:37:50.852 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:37:50.852 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:50.852 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:37:50.852 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 16, 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 16, 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 2, 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 2, 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 3 2020-03-11T22:37:50.853 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.854 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.854 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_stats": [ 2020-03-11T22:37:50.854 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.854 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:37:50.854 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 146028888074 2020-03-11T22:37:50.854 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.854 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 42949672976 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 150323855370 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.855 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_state": [ 2020-03-11T22:37:50.856 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.856 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2020-03-11T22:37:50.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 1 2020-03-11T22:37:50.856 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.856 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_osd": [ 2020-03-11T22:37:50.857 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.857 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:37:50.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:37:50.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:37:50.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 0, 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.858 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.859 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:37:50.859 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 1, 2020-03-11T22:37:50.859 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:37:50.859 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:50.859 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.859 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.859 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [ 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": {} 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: "servicemap": { 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:37:50.860 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:36:13.224373+0000", 2020-03-11T22:37:50.861 INFO:tasks.workunit.client.0.smithi130.stdout: "services": {} 2020-03-11T22:37:50.861 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:37:50.861 INFO:tasks.workunit.client.0.smithi130.stdout: "paxos": { 2020-03-11T22:37:50.861 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:37:50.861 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 306, 2020-03-11T22:37:50.861 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pn": 0, 2020-03-11T22:37:50.861 INFO:tasks.workunit.client.0.smithi130.stdout: "accepted_pn": 0 2020-03-11T22:37:50.862 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:37:50.862 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:37:50.863 INFO:tasks.workunit.client.0.smithi130.stderr:report 252196565 2020-03-11T22:37:50.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:37:50.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:37:50.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:37:50.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:37:50.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:50.864 INFO:tasks.workunit.client.0.smithi130.stdout:writing non-divergent object existing_4 2020-03-11T22:37:51.007 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:51.008 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:37:16.118205+0000 28'100 39:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:37:51.008 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:37:51.008 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:51.009 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:37:51.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:37:51.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:51.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:37:51.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:51.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:51.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:51.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:51.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:51.059 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:51.059 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:51.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:51.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:51.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:51.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:51.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:51.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:51.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:51.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:51.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:37:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:37:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:51.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:51.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888076 2020-03-11T22:37:51.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076' 2020-03-11T22:37:51.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:51.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:51.684 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:37:51.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:37:51.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076 1-' 2020-03-11T22:37:51.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:51.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:51.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:37:51.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076 1- 2-150323855372' 2020-03-11T22:37:51.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:51.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888076 2020-03-11T22:37:51.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:51.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:51.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888076 2020-03-11T22:37:51.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:51.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888076 2020-03-11T22:37:51.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888076' 2020-03-11T22:37:51.825 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 146028888076 2020-03-11T22:37:51.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:52.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888074 -lt 146028888076 2020-03-11T22:37:52.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:53.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:53.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:53.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888077 -lt 146028888076 2020-03-11T22:37:53.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:53.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:37:53.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:53.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:53.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:37:53.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:53.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:37:53.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:37:53.524 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 2020-03-11T22:37:53.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:53.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 2020-03-11T22:37:54.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672976: unary operator expected 2020-03-11T22:37:54.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:54.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855372 2020-03-11T22:37:54.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:54.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:54.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855372 2020-03-11T22:37:54.044 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:54.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:37:54.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2020-03-11T22:37:54.044 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:54.044 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 150323855372 2020-03-11T22:37:54.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855372 -lt 150323855372 2020-03-11T22:37:54.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:54.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:54.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:54.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:54.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:54.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:54.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:54.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:54.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:54.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:54.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:55.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:55.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:55.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:55.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:37:55.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:37:55.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:37:55.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:55.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:37:55.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:55.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:55.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:55.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:55.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:55.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:55.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:55.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:55.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:55.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:55.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:56.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:56.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:56.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:56.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:56.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:56.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:56.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:56.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:56.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:56.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:56.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:56.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:37:56.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:56.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:37:56.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:56.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:56.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:56.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:56.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:56.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:56.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:56.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:57.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:57.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:57.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:57.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:57.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:57.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:57.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:57.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:57.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:57.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:57.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:57.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:57.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:58.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:58.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:58.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:37:58.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:58.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:37:58.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:58.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:58.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:58.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:58.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:58.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:58.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:58.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:58.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:58.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:58.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:58.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:59.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:59.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:59.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:59.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:59.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:59.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:59.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:59.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:59.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:59.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:59.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:59.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:37:59.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:59.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:38:00.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:00.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:00.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:00.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:00.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:00.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:00.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:00.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:00.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:00.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:00.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:01.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:01.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:01.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:01.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:01.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:01.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:01.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:01.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:01.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:01.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:01.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:01.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:38:01.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:01.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:38:03.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:03.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:03.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:03.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:03.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:03.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:03.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:03.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:03.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:03.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:03.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:03.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:04.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:04.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:04.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:04.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:04.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:04.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:04.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:04.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:04.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:04.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:04.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:04.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:38:04.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:04.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:38:07.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:07.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:07.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:07.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:07.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:07.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:07.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:07.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:08.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:08.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:08.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:08.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:08.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:08.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:08.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:08.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:08.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:08.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:08.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:08.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:08.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:09.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:09.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:09.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:38:09.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:09.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:38:15.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:15.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:15.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:15.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:15.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:15.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:15.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:15.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:15.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:15.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:15.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:15.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:16.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:16.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:16.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:16.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:16.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:16.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:16.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:16.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:16.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:16.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:16.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:16.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:38:16.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:16.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:38:24.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:24.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:24.075 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:24.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:24.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:24.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:24.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:24.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:24.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:24.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:24.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:24.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:24.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:24.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:24.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:24.862 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:24.862 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:25.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:25.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:25.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:38:25.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:38:25.739 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:38:25.740 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_fingerprint": "d9851a65-030e-401a-b359-f38a9a034f2f", 2020-03-11T22:38:25.740 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:38:25.740 INFO:tasks.workunit.client.0.smithi130.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:38:25.740 INFO:tasks.workunit.client.0.smithi130.stdout: "timestamp": "2020-03-11T22:38:25.736423+0000", 2020-03-11T22:38:25.740 INFO:tasks.workunit.client.0.smithi130.stdout: "tag": "", 2020-03-11T22:38:25.740 INFO:tasks.workunit.client.0.smithi130.stdout: "health": { 2020-03-11T22:38:25.740 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "HEALTH_WARN", 2020-03-11T22:38:25.741 INFO:tasks.workunit.client.0.smithi130.stdout: "checks": { 2020-03-11T22:38:25.741 INFO:tasks.workunit.client.0.smithi130.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:38:25.741 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:25.741 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:38:25.741 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:38:25.741 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 17 2020-03-11T22:38:25.741 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.742 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [], 2020-03-11T22:38:25.742 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:38:25.742 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.742 INFO:tasks.workunit.client.0.smithi130.stdout: "OSD_DOWN": { 2020-03-11T22:38:25.742 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:25.742 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:38:25.742 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 osds down", 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "osd.1 (root=default,host=smithi130) is down" 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.743 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:38:25.744 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.744 INFO:tasks.workunit.client.0.smithi130.stdout: "PG_DEGRADED": { 2020-03-11T22:38:25.744 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:25.744 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:38:25.744 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:38:25.744 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 2 2020-03-11T22:38:25.744 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.745 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:38:25.745 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.745 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:38:25.745 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.745 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.745 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:38:25.745 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.746 INFO:tasks.workunit.client.0.smithi130.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:38:25.746 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:25.746 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:38:25.746 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:38:25.746 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:38:25.746 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.746 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:38:25.747 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.747 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:38:25.747 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.747 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.747 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:38:25.747 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.747 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: "mutes": [] 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_first_committed": 1, 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_last_committed": 1, 2020-03-11T22:38:25.748 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap": { 2020-03-11T22:38:25.749 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:38:25.749 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:38:25.749 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:36:12.511238Z", 2020-03-11T22:38:25.749 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:36:12.511238Z", 2020-03-11T22:38:25.749 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release": 15, 2020-03-11T22:38:25.749 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:38:25.749 INFO:tasks.workunit.client.0.smithi130.stdout: "features": { 2020-03-11T22:38:25.750 INFO:tasks.workunit.client.0.smithi130.stdout: "persistent": [ 2020-03-11T22:38:25.750 INFO:tasks.workunit.client.0.smithi130.stdout: "kraken", 2020-03-11T22:38:25.750 INFO:tasks.workunit.client.0.smithi130.stdout: "luminous", 2020-03-11T22:38:25.750 INFO:tasks.workunit.client.0.smithi130.stdout: "mimic", 2020-03-11T22:38:25.750 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap-prune", 2020-03-11T22:38:25.750 INFO:tasks.workunit.client.0.smithi130.stdout: "nautilus", 2020-03-11T22:38:25.750 INFO:tasks.workunit.client.0.smithi130.stdout: "octopus" 2020-03-11T22:38:25.751 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.751 INFO:tasks.workunit.client.0.smithi130.stdout: "optional": [] 2020-03-11T22:38:25.751 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.751 INFO:tasks.workunit.client.0.smithi130.stdout: "mons": [ 2020-03-11T22:38:25.751 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: "rank": 0, 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "a", 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:38:25.752 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 0 2020-03-11T22:38:25.753 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.753 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.753 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.753 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:38:25.753 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:38:25.753 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 0, 2020-03-11T22:38:25.753 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 0 2020-03-11T22:38:25.754 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.754 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.754 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.754 INFO:tasks.workunit.client.0.smithi130.stdout: "quorum": [ 2020-03-11T22:38:25.754 INFO:tasks.workunit.client.0.smithi130.stdout: 0 2020-03-11T22:38:25.754 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.754 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap": { 2020-03-11T22:38:25.755 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 39, 2020-03-11T22:38:25.755 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:38:25.755 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:36:13.224040+0000", 2020-03-11T22:38:25.755 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:37:15.925422+0000", 2020-03-11T22:38:25.755 INFO:tasks.workunit.client.0.smithi130.stdout: "last_up_change": "2020-03-11T22:37:14.287519+0000", 2020-03-11T22:38:25.755 INFO:tasks.workunit.client.0.smithi130.stdout: "last_in_change": "2020-03-11T22:37:15.698163+0000", 2020-03-11T22:38:25.755 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_num": 5800384, 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_set": [ 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "nodown", 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "noin", 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "noout", 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "pglog_hardlimit", 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snapdirs", 2020-03-11T22:38:25.756 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_deletes", 2020-03-11T22:38:25.757 INFO:tasks.workunit.client.0.smithi130.stdout: "sortbitwise" 2020-03-11T22:38:25.757 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.757 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_version": 7, 2020-03-11T22:38:25.757 INFO:tasks.workunit.client.0.smithi130.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:38:25.757 INFO:tasks.workunit.client.0.smithi130.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:38:25.757 INFO:tasks.workunit.client.0.smithi130.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:38:25.757 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_snapshot": "", 2020-03-11T22:38:25.758 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_max": 1, 2020-03-11T22:38:25.758 INFO:tasks.workunit.client.0.smithi130.stdout: "max_osd": 3, 2020-03-11T22:38:25.758 INFO:tasks.workunit.client.0.smithi130.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:38:25.758 INFO:tasks.workunit.client.0.smithi130.stdout: "min_compat_client": "jewel", 2020-03-11T22:38:25.758 INFO:tasks.workunit.client.0.smithi130.stdout: "require_osd_release": "octopus", 2020-03-11T22:38:25.758 INFO:tasks.workunit.client.0.smithi130.stdout: "pools": [ 2020-03-11T22:38:25.758 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.759 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:38:25.759 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_name": "test", 2020-03-11T22:38:25.759 INFO:tasks.workunit.client.0.smithi130.stdout: "create_time": "2020-03-11T22:36:34.399712+0000", 2020-03-11T22:38:25.759 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": 1, 2020-03-11T22:38:25.759 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_names": "hashpspool", 2020-03-11T22:38:25.759 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:38:25.760 INFO:tasks.workunit.client.0.smithi130.stdout: "size": 3, 2020-03-11T22:38:25.760 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 2, 2020-03-11T22:38:25.760 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_rule": 0, 2020-03-11T22:38:25.760 INFO:tasks.workunit.client.0.smithi130.stdout: "object_hash": 2, 2020-03-11T22:38:25.760 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:38:25.760 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num": 1, 2020-03-11T22:38:25.760 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num": 1, 2020-03-11T22:38:25.761 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num_target": 1, 2020-03-11T22:38:25.761 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_target": 1, 2020-03-11T22:38:25.761 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_pending": 1, 2020-03-11T22:38:25.761 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pg_merge_meta": { 2020-03-11T22:38:25.761 INFO:tasks.workunit.client.0.smithi130.stdout: "source_pgid": "0.0", 2020-03-11T22:38:25.761 INFO:tasks.workunit.client.0.smithi130.stdout: "ready_epoch": 0, 2020-03-11T22:38:25.761 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 0, 2020-03-11T22:38:25.762 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 0, 2020-03-11T22:38:25.762 INFO:tasks.workunit.client.0.smithi130.stdout: "source_version": "0'0", 2020-03-11T22:38:25.762 INFO:tasks.workunit.client.0.smithi130.stdout: "target_version": "0'0" 2020-03-11T22:38:25.762 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.762 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "28", 2020-03-11T22:38:25.762 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend": "0", 2020-03-11T22:38:25.763 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:38:25.763 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:38:25.763 INFO:tasks.workunit.client.0.smithi130.stdout: "auid": 0, 2020-03-11T22:38:25.763 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:38:25.763 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_seq": 0, 2020-03-11T22:38:25.763 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_epoch": 0, 2020-03-11T22:38:25.763 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_snaps": [], 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps": "[]", 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_bytes": 0, 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_objects": 0, 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "tiers": [], 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "tier_of": -1, 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "read_tier": -1, 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "write_tier": -1, 2020-03-11T22:38:25.764 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_mode": "none", 2020-03-11T22:38:25.765 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_bytes": 0, 2020-03-11T22:38:25.765 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_objects": 0, 2020-03-11T22:38:25.765 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:38:25.765 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:38:25.765 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:38:25.765 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_flush_age": 0, 2020-03-11T22:38:25.765 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_evict_age": 0, 2020-03-11T22:38:25.766 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profile": "", 2020-03-11T22:38:25.766 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_params": { 2020-03-11T22:38:25.766 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "none" 2020-03-11T22:38:25.766 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.766 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_period": 0, 2020-03-11T22:38:25.766 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_count": 0, 2020-03-11T22:38:25.766 INFO:tasks.workunit.client.0.smithi130.stdout: "use_gmt_hitset": true, 2020-03-11T22:38:25.767 INFO:tasks.workunit.client.0.smithi130.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:38:25.767 INFO:tasks.workunit.client.0.smithi130.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:38:25.767 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:38:25.767 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:38:25.767 INFO:tasks.workunit.client.0.smithi130.stdout: "grade_table": [], 2020-03-11T22:38:25.767 INFO:tasks.workunit.client.0.smithi130.stdout: "stripe_width": 0, 2020-03-11T22:38:25.767 INFO:tasks.workunit.client.0.smithi130.stdout: "expected_num_objects": 0, 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: "fast_read": false, 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: "options": {}, 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: "application_metadata": {} 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: "osds": [ 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.768 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:38:25.769 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "1b8f82d0-d42b-4d93-ab95-f22ad2bc035e", 2020-03-11T22:38:25.769 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:38:25.769 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:38:25.769 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:38:25.769 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:38:25.769 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 5, 2020-03-11T22:38:25.769 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 28, 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 34, 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 38, 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 29, 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.770 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.771 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:38:25.771 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.771 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.771 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.771 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.771 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:38:25.771 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.772 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:38:25.773 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.773 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.773 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.773 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.773 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:38:25.773 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.773 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.774 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.774 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.774 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:25.774 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.774 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.774 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.774 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.775 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.776 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.776 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:25.776 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.776 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.776 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.776 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:38:25.776 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.777 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.777 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.777 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.777 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:38:25.777 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30293 2020-03-11T22:38:25.777 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.777 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.778 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.778 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6803/30293", 2020-03-11T22:38:25.778 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6805/30293", 2020-03-11T22:38:25.778 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30293", 2020-03-11T22:38:25.778 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30293", 2020-03-11T22:38:25.778 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:38:25.779 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:38:25.779 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:38:25.779 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.779 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.779 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.779 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "e938a584-5c23-4b9f-8a94-a38336bc98c0", 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 0, 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 0, 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 0, 2020-03-11T22:38:25.780 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 10, 2020-03-11T22:38:25.781 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 22, 2020-03-11T22:38:25.781 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 30, 2020-03-11T22:38:25.781 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:38:25.781 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:38:25.781 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.781 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:25.782 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.783 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.783 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.783 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.783 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:38:25.783 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.783 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.783 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.784 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:25.784 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.784 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.784 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.784 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.784 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:25.784 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.785 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:25.786 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.786 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.786 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.786 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.786 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:25.786 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.786 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:25.787 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.788 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.788 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.788 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.788 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:25.788 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 25860 2020-03-11T22:38:25.788 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.788 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.789 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.789 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/25860", 2020-03-11T22:38:25.789 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/25860", 2020-03-11T22:38:25.789 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25860", 2020-03-11T22:38:25.789 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25860", 2020-03-11T22:38:25.789 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:38:25.789 INFO:tasks.workunit.client.0.smithi130.stdout: "exists" 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "ad75aea1-426e-4305-9d25-2742006cffd6", 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:38:25.790 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:38:25.791 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:38:25.791 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 15, 2020-03-11T22:38:25.791 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 28, 2020-03-11T22:38:25.791 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 35, 2020-03-11T22:38:25.791 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 0, 2020-03-11T22:38:25.791 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 31, 2020-03-11T22:38:25.791 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.792 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.793 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.793 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:25.794 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.794 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.794 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.794 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.794 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:38:25.794 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.795 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.795 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.795 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:25.795 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.795 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.795 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.795 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.796 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:25.796 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.796 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.796 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.796 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.797 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:25.797 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.797 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.797 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.797 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:25.797 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.797 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.798 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.798 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.798 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:25.798 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.798 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.798 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.798 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.799 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:25.799 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:38:25.799 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.799 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:38:25.799 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:25.799 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.800 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.800 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.800 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:38:25.800 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:25.800 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 30487 2020-03-11T22:38:25.800 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.800 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.801 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.801 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/30487", 2020-03-11T22:38:25.801 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/30487", 2020-03-11T22:38:25.801 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30487", 2020-03-11T22:38:25.801 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30487", 2020-03-11T22:38:25.801 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:38:25.801 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_xinfo": [ 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:38:25.802 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:37:08.235832+0000", 2020-03-11T22:38:25.803 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:38:25.803 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:38:25.803 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:38:25.803 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:38:25.803 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:20.177768+0000", 2020-03-11T22:38:25.803 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:38:25.803 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.804 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.804 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:38:25.804 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:37:08.733445+0000", 2020-03-11T22:38:25.804 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:38:25.804 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:38:25.804 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:38:25.804 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:26.520312+0000", 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:37:09.228952+0000", 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:38:25.805 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:38:25.806 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:38:25.806 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:38:25.806 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:36:32.531833+0000", 2020-03-11T22:38:25.806 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:38:25.806 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.806 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.807 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap": [], 2020-03-11T22:38:25.807 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap_items": [], 2020-03-11T22:38:25.807 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_temp": [], 2020-03-11T22:38:25.807 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_temp": [], 2020-03-11T22:38:25.807 INFO:tasks.workunit.client.0.smithi130.stdout: "blacklist": {}, 2020-03-11T22:38:25.807 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profiles": { 2020-03-11T22:38:25.808 INFO:tasks.workunit.client.0.smithi130.stdout: "default": { 2020-03-11T22:38:25.808 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:38:25.808 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "2", 2020-03-11T22:38:25.808 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:38:25.808 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van" 2020-03-11T22:38:25.808 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.808 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.809 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps_queue": [], 2020-03-11T22:38:25.809 INFO:tasks.workunit.client.0.smithi130.stdout: "new_removed_snaps": [], 2020-03-11T22:38:25.809 INFO:tasks.workunit.client.0.smithi130.stdout: "new_purged_snaps": [], 2020-03-11T22:38:25.809 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_node_flags": {}, 2020-03-11T22:38:25.809 INFO:tasks.workunit.client.0.smithi130.stdout: "device_class_flags": {} 2020-03-11T22:38:25.809 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.809 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_metadata": [ 2020-03-11T22:38:25.810 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.810 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:38:25.810 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:38:25.810 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6804/30293,v1:127.0.0.1:6805/30293]", 2020-03-11T22:38:25.810 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:38:25.810 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:38:25.810 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:38:25.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:25.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:25.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:25.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:25.812 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:38:25.812 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:25.812 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:25.812 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:38:25.813 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6802/30293,v1:127.0.0.1:6803/30293]", 2020-03-11T22:38:25.814 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:38:25.814 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30293,v1:127.0.0.1:6809/30293]", 2020-03-11T22:38:25.814 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30293,v1:127.0.0.1:6807/30293]", 2020-03-11T22:38:25.814 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:38:25.814 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:38:25.814 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:25.814 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:25.815 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:38:25.816 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.816 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.816 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:38:25.816 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:38:25.816 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/25860,v1:127.0.0.1:6813/25860]", 2020-03-11T22:38:25.816 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:38:25.816 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:38:25.817 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:25.817 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:25.817 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:25.817 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:25.818 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:25.819 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:38:25.819 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:25.819 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:25.819 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:38:25.819 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:38:25.819 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:38:25.819 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:25.820 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:38:25.820 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:38:25.820 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:25.820 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:38:25.820 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/25860,v1:127.0.0.1:6811/25860]", 2020-03-11T22:38:25.820 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:38:25.820 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25860,v1:127.0.0.1:6817/25860]", 2020-03-11T22:38:25.821 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25860,v1:127.0.0.1:6815/25860]", 2020-03-11T22:38:25.821 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:38:25.821 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:38:25.821 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:25.821 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:25.821 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:25.821 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.822 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/30487,v1:127.0.0.1:6813/30487]", 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:25.823 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:25.824 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:25.824 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:25.824 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:25.824 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:25.824 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:38:25.824 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:25.824 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:25.825 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:25.825 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:25.825 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:38:25.825 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:25.825 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:25.825 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:38:25.825 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:38:25.826 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:38:25.826 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:25.826 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:38:25.826 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:38:25.826 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:25.826 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:38:25.826 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/30487,v1:127.0.0.1:6811/30487]", 2020-03-11T22:38:25.827 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:38:25.827 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30487,v1:127.0.0.1:6817/30487]", 2020-03-11T22:38:25.827 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30487,v1:127.0.0.1:6815/30487]", 2020-03-11T22:38:25.827 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:38:25.827 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:38:25.828 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:25.828 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:25.828 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:25.828 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:38:25.828 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:25.828 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:25.828 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:38:25.829 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:38:25.829 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:25.829 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:38:25.829 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.829 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.829 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_first_committed": 1, 2020-03-11T22:38:25.829 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_last_committed": 39, 2020-03-11T22:38:25.830 INFO:tasks.workunit.client.0.smithi130.stdout: "crushmap": { 2020-03-11T22:38:25.830 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": [ 2020-03-11T22:38:25.830 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.830 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:38:25.830 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.0", 2020-03-11T22:38:25.830 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.1", 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.831 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:38:25.832 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.2", 2020-03-11T22:38:25.832 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:38:25.832 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.832 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.832 INFO:tasks.workunit.client.0.smithi130.stdout: "types": [ 2020-03-11T22:38:25.832 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.832 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 0, 2020-03-11T22:38:25.833 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd" 2020-03-11T22:38:25.833 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.833 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.833 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:38:25.833 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "host" 2020-03-11T22:38:25.833 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.833 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.834 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 2, 2020-03-11T22:38:25.834 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "chassis" 2020-03-11T22:38:25.834 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.834 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.834 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 3, 2020-03-11T22:38:25.834 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "rack" 2020-03-11T22:38:25.834 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.835 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.835 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 4, 2020-03-11T22:38:25.835 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "row" 2020-03-11T22:38:25.835 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.835 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.835 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 5, 2020-03-11T22:38:25.835 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pdu" 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 6, 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pod" 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 7, 2020-03-11T22:38:25.836 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "room" 2020-03-11T22:38:25.837 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.837 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.837 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 8, 2020-03-11T22:38:25.837 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "datacenter" 2020-03-11T22:38:25.837 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.837 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.837 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 9, 2020-03-11T22:38:25.838 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "zone" 2020-03-11T22:38:25.838 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.838 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.838 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 10, 2020-03-11T22:38:25.838 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "region" 2020-03-11T22:38:25.839 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.839 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.839 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:38:25.839 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "root" 2020-03-11T22:38:25.839 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.839 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.839 INFO:tasks.workunit.client.0.smithi130.stdout: "buckets": [ 2020-03-11T22:38:25.840 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.840 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -1, 2020-03-11T22:38:25.840 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default", 2020-03-11T22:38:25.840 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:38:25.840 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:38:25.840 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:38:25.840 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:38:25.841 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:38:25.841 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:38:25.841 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.841 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:38:25.841 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:38:25.841 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:38:25.842 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.842 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.842 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.842 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.842 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -2, 2020-03-11T22:38:25.842 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default~hdd", 2020-03-11T22:38:25.842 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:38:25.843 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:38:25.843 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:38:25.843 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:38:25.843 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:38:25.843 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:38:25.843 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.843 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:38:25.844 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:38:25.844 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:38:25.844 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.844 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.844 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.844 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.844 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130", 2020-03-11T22:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:38:25.846 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:38:25.847 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:38:25.847 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.847 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.847 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:38:25.847 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:38:25.847 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:38:25.847 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.848 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.848 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.848 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.848 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:38:25.848 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130~hdd", 2020-03-11T22:38:25.849 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:38:25.849 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:38:25.849 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:38:25.849 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:38:25.849 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:38:25.849 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:38:25.850 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:38:25.851 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:38:25.851 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.851 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.851 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:38:25.851 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:38:25.851 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:38:25.851 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.852 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.852 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.852 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.852 INFO:tasks.workunit.client.0.smithi130.stdout: "rules": [ 2020-03-11T22:38:25.852 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.852 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 0, 2020-03-11T22:38:25.852 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "replicated_rule", 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 0, 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 1, 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 10, 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:38:25.853 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:38:25.854 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:38:25.854 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.854 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.854 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_firstn", 2020-03-11T22:38:25.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:38:25.854 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:38:25.854 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: "tunables": { 2020-03-11T22:38:25.855 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_tries": 0, 2020-03-11T22:38:25.856 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:38:25.856 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_total_tries": 50, 2020-03-11T22:38:25.856 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:38:25.856 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:38:25.856 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_stable": 1, 2020-03-11T22:38:25.856 INFO:tasks.workunit.client.0.smithi130.stdout: "straw_calc_version": 1, 2020-03-11T22:38:25.856 INFO:tasks.workunit.client.0.smithi130.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:38:25.857 INFO:tasks.workunit.client.0.smithi130.stdout: "profile": "jewel", 2020-03-11T22:38:25.857 INFO:tasks.workunit.client.0.smithi130.stdout: "optimal_tunables": 1, 2020-03-11T22:38:25.857 INFO:tasks.workunit.client.0.smithi130.stdout: "legacy_tunables": 0, 2020-03-11T22:38:25.857 INFO:tasks.workunit.client.0.smithi130.stdout: "minimum_required_version": "jewel", 2020-03-11T22:38:25.857 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables": 1, 2020-03-11T22:38:25.857 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables2": 1, 2020-03-11T22:38:25.857 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v2_rules": 0, 2020-03-11T22:38:25.858 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables3": 1, 2020-03-11T22:38:25.858 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v3_rules": 0, 2020-03-11T22:38:25.858 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v4_buckets": 1, 2020-03-11T22:38:25.858 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables5": 1, 2020-03-11T22:38:25.858 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v5_rules": 0 2020-03-11T22:38:25.859 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.859 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_args": {} 2020-03-11T22:38:25.859 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.859 INFO:tasks.workunit.client.0.smithi130.stdout: "fsmap": { 2020-03-11T22:38:25.859 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:38:25.859 INFO:tasks.workunit.client.0.smithi130.stdout: "default_fscid": -1, 2020-03-11T22:38:25.859 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": { 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": {}, 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "ro_compat": {}, 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "incompat": { 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_1": "base v0.20", 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:38:25.860 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:38:25.861 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:38:25.861 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_8": "no anchor table", 2020-03-11T22:38:25.861 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_9": "file layout v2", 2020-03-11T22:38:25.861 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:38:25.861 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.861 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.861 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_flags": { 2020-03-11T22:38:25.862 INFO:tasks.workunit.client.0.smithi130.stdout: "enable_multiple": false, 2020-03-11T22:38:25.862 INFO:tasks.workunit.client.0.smithi130.stdout: "ever_enabled_multiple": false 2020-03-11T22:38:25.862 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.862 INFO:tasks.workunit.client.0.smithi130.stdout: "standbys": [], 2020-03-11T22:38:25.862 INFO:tasks.workunit.client.0.smithi130.stdout: "filesystems": [] 2020-03-11T22:38:25.862 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.862 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:38:25.863 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:38:25.863 INFO:tasks.workunit.client.0.smithi130.stdout: "auth": { 2020-03-11T22:38:25.863 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:38:25.863 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 5, 2020-03-11T22:38:25.863 INFO:tasks.workunit.client.0.smithi130.stdout: "num_secrets": 3 2020-03-11T22:38:25.863 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.863 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:38:25.864 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_active": 1, 2020-03-11T22:38:25.864 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_unknown": 0, 2020-03-11T22:38:25.864 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osd": 3, 2020-03-11T22:38:25.864 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_sum": { 2020-03-11T22:38:25.864 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:38:25.864 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 47947, 2020-03-11T22:38:25.864 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:38:25.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:38:25.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:38:25.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:25.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:38:25.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:25.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:25.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:25.866 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:25.866 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:38:25.866 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:38:25.866 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:38:25.866 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:38:25.866 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:38:25.867 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:25.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:25.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:38:25.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:38:25.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:38:25.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:38:25.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:38:25.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:25.869 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:25.869 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:25.869 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:25.869 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:25.870 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:25.870 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:25.870 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:25.870 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:25.870 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:38:25.870 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:38:25.870 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.871 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:38:25.871 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:38:25.871 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:38:25.871 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:38:25.871 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 0, 2020-03-11T22:38:25.871 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 0, 2020-03-11T22:38:25.872 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:38:25.872 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:25.872 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:38:25.872 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:38:25.872 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:38:25.872 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.873 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 17, 2020-03-11T22:38:25.873 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 17, 2020-03-11T22:38:25.873 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 2, 2020-03-11T22:38:25.873 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 2, 2020-03-11T22:38:25.873 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 0 2020-03-11T22:38:25.873 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.873 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum": { 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:38:25.874 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3179072, 2020-03-11T22:38:25.875 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 33024, 2020-03-11T22:38:25.875 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 13, 2020-03-11T22:38:25.875 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145714, 2020-03-11T22:38:25.875 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393728, 2020-03-11T22:38:25.875 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:38:25.875 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:38:25.875 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318867177472, 2020-03-11T22:38:25.876 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:25.876 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 33816576, 2020-03-11T22:38:25.876 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 401385, 2020-03-11T22:38:25.876 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:38:25.876 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:25.876 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:38:25.876 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 14090, 2020-03-11T22:38:25.877 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221211382 2020-03-11T22:38:25.877 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.877 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:38:25.877 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:25.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:25.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:38:25.877 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:25.878 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:38:25.879 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.879 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:38:25.879 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:38:25.879 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum_by_class": { 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "hdd": { 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:25.880 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:25.881 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:38:25.881 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3179072, 2020-03-11T22:38:25.881 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 33024, 2020-03-11T22:38:25.881 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 13, 2020-03-11T22:38:25.881 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145714, 2020-03-11T22:38:25.881 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393728, 2020-03-11T22:38:25.881 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318867177472, 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 33816576, 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 401385, 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:25.882 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:38:25.883 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 14090, 2020-03-11T22:38:25.883 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221211382 2020-03-11T22:38:25.883 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.883 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:38:25.883 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:25.883 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:25.883 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:25.884 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:25.885 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:38:25.885 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.885 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:38:25.885 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:38:25.885 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.885 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.885 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_stats": [ 2020-03-11T22:38:25.886 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.886 INFO:tasks.workunit.client.0.smithi130.stdout: "poolid": 1, 2020-03-11T22:38:25.886 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:38:25.886 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:38:25.886 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 47947, 2020-03-11T22:38:25.886 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:38:25.886 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:25.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:25.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:25.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:25.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:25.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:38:25.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:25.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:25.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:38:25.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:38:25.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:38:25.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:38:25.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:38:25.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:25.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:25.891 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:25.891 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:25.891 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:25.891 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:25.891 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:25.891 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:38:25.892 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 19529728, 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 142522, 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:38:25.893 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.894 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 17, 2020-03-11T22:38:25.894 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 17, 2020-03-11T22:38:25.894 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 2, 2020-03-11T22:38:25.894 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 2, 2020-03-11T22:38:25.894 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 3 2020-03-11T22:38:25.894 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.894 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.895 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_stats": [ 2020-03-11T22:38:25.895 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.895 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:38:25.895 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 146028888083 2020-03-11T22:38:25.895 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.895 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.896 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:38:25.896 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 42949672976 2020-03-11T22:38:25.896 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.896 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.896 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:38:25.896 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 150323855379 2020-03-11T22:38:25.896 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_state": [ 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 1 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.897 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_osd": [ 2020-03-11T22:38:25.898 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.898 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:38:25.898 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:38:25.898 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:38:25.898 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:25.898 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.898 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.899 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:38:25.899 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:38:25.899 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 0, 2020-03-11T22:38:25.899 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:25.899 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.899 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.899 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:38:25.900 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 1, 2020-03-11T22:38:25.900 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:38:25.900 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:25.900 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.900 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.901 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [ 2020-03-11T22:38:25.901 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:38:25.901 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:38:25.901 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": {} 2020-03-11T22:38:25.901 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.901 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:38:25.901 INFO:tasks.workunit.client.0.smithi130.stdout: "servicemap": { 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:36:13.224373+0000", 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: "services": {} 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: "paxos": { 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 383, 2020-03-11T22:38:25.902 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pn": 0, 2020-03-11T22:38:25.903 INFO:tasks.workunit.client.0.smithi130.stdout: "accepted_pn": 0 2020-03-11T22:38:25.903 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:38:25.903 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:38:25.904 INFO:tasks.workunit.client.0.smithi130.stderr:report 2421884532 2020-03-11T22:38:25.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:38:25.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:38:25.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:25.904 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery 2020-03-11T22:38:26.070 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:26.070 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:16.118205+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:38:26.070 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:38:26.070 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:26.071 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:38:26.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:26.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:38:26.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:26.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:26.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:26.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:26.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:38:26.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:38:26.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:26.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:38:26.172 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:26.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:26.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:38:26.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:26.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:26.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:26.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:26.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:38:26.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:38:26.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:26.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:38:26.273 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:26.280 INFO:tasks.workunit.client.0.smithi130.stdout:revive divergent 1 2020-03-11T22:38:26.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:38:26.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:26.600 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:26.601 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:16.118205+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:38:26.601 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:38:26.601 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:26.602 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:38:26.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:38:27.228 INFO:tasks.workunit.client.0.smithi130.stderr:noup is set 2020-03-11T22:38:27.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:38:27.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:27.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:27.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:27.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:27.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:27.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:38:27.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:38:27.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:38:27.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:27.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:27.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:27.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:27.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:27.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:27.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:27.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:27.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:27.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:27.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:27.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:27.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:27.244 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:38:27.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:27.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:27.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:27.265 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:27.261+0000 7f581f462ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:27.266 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:27.263+0000 7f581f462ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:27.267 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:27.264+0000 7f581f462ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:27.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:38:27.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:38:27.783 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:27.780+0000 7f581f462ec0 -1 Falling back to public interface 2020-03-11T22:38:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:28.621+0000 7f581f462ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:38:32.591 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery divergent 2020-03-11T22:38:32.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:38:32.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:32.921 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:32.921 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:37:16.118205+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:38:32.922 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:38:32.922 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:32.922 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:38:32.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:38:32.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:38:32.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:32.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:38:33.033 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:33.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:38:33.695 INFO:tasks.workunit.client.0.smithi130.stderr:noup is unset 2020-03-11T22:38:33.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:38:33.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:33.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:33.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:33.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:33.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:33.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:33.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:33.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:33.704 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:38:34.054 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 34 up_thru 38 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30293,v1:127.0.0.1:6803/30293] [v2:127.0.0.1:6804/30293,v1:127.0.0.1:6805/30293] exists,up 1b8f82d0-d42b-4d93-ab95-f22ad2bc035e 2020-03-11T22:38:34.054 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:38:34.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:34.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:34.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:34.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:38:34.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:34.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:34.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:34.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:34.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:34.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:34.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:34.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:34.402 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33034,v1:127.0.0.1:6819/33034] [v2:127.0.0.1:6820/33034,v1:127.0.0.1:6821/33034] exists,up e938a584-5c23-4b9f-8a94-a38336bc98c0 2020-03-11T22:38:34.402 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:38:34.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:34.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:34.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:34.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:38:34.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:34.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:34.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:34.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:34.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:34.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:34.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:34.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:34.745 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30487,v1:127.0.0.1:6811/30487] [v2:127.0.0.1:6812/30487,v1:127.0.0.1:6813/30487] exists,up ad75aea1-426e-4305-9d25-2742006cffd6 2020-03-11T22:38:34.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:34.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:34.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:34.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:35.066 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:35.066 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:38:33.543994+0000 28'106 43:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:38:35.066 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:38:35.066 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:35.067 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:38:35.076 INFO:tasks.workunit.client.0.smithi130.stdout:wait for peering 2020-03-11T22:38:35.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:38:35.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:35.397 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:35.398 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:38:33.543994+0000 28'106 43:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:38:35.398 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:38:35.398 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:35.399 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:38:35.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:38:44.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:38:44.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:44.155 INFO:tasks.workunit.client.0.smithi130.stdout:killing divergent 1 2020-03-11T22:38:44.484 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:44.485 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:38:33.878844+0000 39'101 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:38:44.485 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:38:44.485 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:44.486 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:38:44.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:38:44.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:44.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:44.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:44.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:44.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:44.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:44.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17424.out 2020-03-11T22:38:44.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17424.out 2020-03-11T22:38:44.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:38:44.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:38:44.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:38:44.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:38:44.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:38:44.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17424.out 2020-03-11T22:38:45.129 INFO:tasks.workunit.client.0.smithi130.stderr:Exporting 1.0 info 1.0( v 44'102 lc 0'0 (28'90,44'102] local-lis/les=43/44 n=101 ec=22/22 lis/c=43/22 les/c/f=44/23/0 sis=43 pruub=10.243172464s) 2020-03-11T22:38:45.142 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:38:45.161 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:38:45.179 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:38:45.189 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:38:45.189 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:38:45.195 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:38:45.196 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:38:45.197 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:38:45.197 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:38:45.198 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:38:45.198 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:38:45.199 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:38:45.206 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:38:45.206 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:38:45.207 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:38:45.207 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:38:45.216 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:38:45.217 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:38:45.218 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:38:45.218 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:38:45.218 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:38:45.221 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:38:45.226 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:38:45.232 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:38:45.232 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:38:45.233 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:38:45.233 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:38:45.234 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:38:45.235 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:38:45.235 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:38:45.236 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:38:45.236 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:38:45.237 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:38:45.237 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:38:45.237 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:38:45.238 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:38:45.239 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:38:45.247 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:38:45.247 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:38:45.248 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:38:45.248 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:38:45.251 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:38:45.252 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:38:45.252 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:38:45.253 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:38:45.253 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:38:45.256 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:38:45.257 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:38:45.257 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:38:45.258 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:38:45.258 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:38:45.259 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:38:45.259 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:38:45.260 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:38:45.260 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:38:45.260 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:38:45.261 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:38:45.264 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:38:45.264 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:38:45.265 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:38:45.266 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:38:45.266 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:38:45.267 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:38:45.267 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:38:45.268 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:38:45.268 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:38:45.269 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:38:45.270 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:38:45.270 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:38:45.288 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:38:45.289 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:38:45.289 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:38:45.290 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:38:45.290 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:38:45.291 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:38:45.291 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:38:45.299 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:38:45.299 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:38:45.300 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:38:45.300 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:38:45.301 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:38:45.301 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:38:45.302 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:38:45.302 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:38:45.303 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:38:45.304 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:38:45.305 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:38:45.305 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:38:45.306 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:38:45.306 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:38:45.318 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:38:45.318 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:38:45.318 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:38:45.319 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:38:45.319 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:38:45.319 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:38:45.319 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:38:45.319 INFO:tasks.workunit.client.0.smithi130.stderr:Export successful 2020-03-11T22:38:45.320 INFO:tasks.workunit.client.0.smithi130.stdout: marking collection for removal 2020-03-11T22:38:45.320 INFO:tasks.workunit.client.0.smithi130.stdout:setting '_remove' omap key 2020-03-11T22:38:45.320 INFO:tasks.workunit.client.0.smithi130.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:38:45.445 INFO:tasks.workunit.client.0.smithi130.stderr:Remove successful 2020-03-11T22:38:45.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17424.out 2020-03-11T22:38:45.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:38:45.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:38:45.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:38:45.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:38:45.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:38:45.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17424.out 2020-03-11T22:38:46.281 INFO:tasks.workunit.client.0.smithi130.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={22=1}} deleted_pools ) 2020-03-11T22:38:46.281 INFO:tasks.workunit.client.0.smithi130.stdout:Importing pgid 1.0 2020-03-11T22:38:46.281 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:38:46.282 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.335 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:38:46.336 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.344 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:38:46.344 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.352 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:38:46.352 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.360 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:38:46.361 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.369 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:38:46.369 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.377 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:38:46.377 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.385 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:38:46.385 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.394 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:38:46.394 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.402 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:38:46.402 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.411 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:38:46.411 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.419 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:38:46.419 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.427 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:38:46.427 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.435 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:38:46.436 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.479 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:38:46.479 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.516 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:38:46.516 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.524 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:38:46.524 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.533 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:38:46.533 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.541 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:38:46.541 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.549 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:38:46.550 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.558 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:38:46.558 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.566 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:38:46.566 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.574 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:38:46.575 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.583 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:38:46.583 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.591 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:38:46.591 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.600 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:38:46.600 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.608 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:38:46.608 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.625 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:38:46.625 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.679 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:38:46.680 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.686 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:38:46.686 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.694 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:38:46.694 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.702 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:38:46.702 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.711 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:38:46.711 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.719 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:38:46.719 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.727 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:38:46.728 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.736 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:38:46.736 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.744 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:38:46.744 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.752 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:38:46.753 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.761 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:38:46.761 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.769 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:38:46.770 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.778 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:38:46.778 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.813 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:38:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.819 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:38:46.819 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.828 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:38:46.828 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.836 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:38:46.836 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.844 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:38:46.845 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.853 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:38:46.853 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.861 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:38:46.861 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.869 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:38:46.870 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.878 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:38:46.878 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.886 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:38:46.886 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.895 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:38:46.895 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.903 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:38:46.903 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.911 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:38:46.911 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.920 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:38:46.920 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.963 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:38:46.963 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.970 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:38:46.970 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.978 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:38:46.978 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.986 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:38:46.987 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:46.995 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:38:46.995 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.003 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:38:47.003 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.013 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:38:47.013 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.020 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:38:47.020 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.029 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:38:47.029 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.037 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:38:47.037 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.080 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:38:47.080 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.140 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:38:47.140 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.153 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:38:47.153 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.188 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:38:47.188 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.195 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:38:47.195 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.203 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:38:47.203 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.212 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:38:47.212 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.220 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:38:47.220 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.228 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:38:47.228 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.237 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:38:47.237 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.245 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:38:47.245 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.253 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:38:47.254 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.262 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:38:47.262 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.270 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:38:47.270 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.278 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:38:47.279 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.287 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:38:47.287 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.295 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:38:47.295 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.330 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:38:47.330 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.337 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:38:47.337 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.345 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:38:47.345 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.370 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:38:47.371 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.388 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:38:47.388 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.413 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:38:47.413 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.430 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:38:47.430 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.437 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:38:47.438 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.446 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:38:47.446 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.454 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:38:47.454 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.462 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:38:47.462 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.470 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:38:47.471 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.479 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:38:47.479 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.487 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:38:47.487 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.522 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:38:47.522 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:38:47.529 INFO:tasks.workunit.client.0.smithi130.stdout:write_pg epoch 44 info 1.0( v 44'102 lc 0'0 (28'90,44'102] local-lis/les=43/44 n=101 ec=22/22 lis/c=43/22 les/c/f=44/23/0 sis=43 pruub=10.243172464s) 2020-03-11T22:38:47.537 INFO:tasks.workunit.client.0.smithi130.stdout:Import successful 2020-03-11T22:38:48.061 INFO:tasks.workunit.client.0.smithi130.stdout:reviving divergent 1 2020-03-11T22:38:48.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:38:48.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:48.391 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:48.392 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:38:33.878844+0000 39'101 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:38:48.392 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:38:48.392 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:48.393 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:38:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:38:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:48.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:48.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:48.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:48.407 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:48.407 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:48.407 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:38:48.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:38:48.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:38:48.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:48.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:48.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:48.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:48.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:48.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:48.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:48.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:48.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:48.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:48.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:48.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:48.410 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:38:48.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:48.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:48.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:48.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:48.430 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:48.428+0000 7f498a38cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:48.432 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:48.430+0000 7f498a38cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:48.434 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:48.431+0000 7f498a38cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:48.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:38:48.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:48.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:48.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:48.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:48.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:48.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:48.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:48.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:48.750 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:38:48.952 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:48.950+0000 7f498a38cec0 -1 Falling back to public interface 2020-03-11T22:38:49.097 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33034,v1:127.0.0.1:6819/33034] [v2:127.0.0.1:6820/33034,v1:127.0.0.1:6821/33034] exists,up e938a584-5c23-4b9f-8a94-a38336bc98c0 2020-03-11T22:38:49.097 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:38:49.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:49.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:49.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:49.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:49.452 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33034,v1:127.0.0.1:6819/33034] [v2:127.0.0.1:6820/33034,v1:127.0.0.1:6821/33034] exists,up e938a584-5c23-4b9f-8a94-a38336bc98c0 2020-03-11T22:38:49.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:49.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:49.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:49.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:38:49.705 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:38:49.702+0000 7f498a38cec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-11T22:39:09.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:39:09.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:39:09.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:39:09.454 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:09.455 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:09.455 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:09.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:39:09.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:39:09.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:39:09.551 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:39:09.551 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [], 2020-03-11T22:39:09.552 INFO:tasks.workunit.client.0.smithi130.stdout: "num_ops": 0 2020-03-11T22:39:09.552 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:39:09.559 INFO:tasks.workunit.client.0.smithi130.stdout:allowing recovery 2020-03-11T22:39:09.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:39:09.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:09.882 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:09.882 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:38:50.900443+0000 44'102 47:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:39:09.882 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:39:09.883 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:09.884 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:39:09.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:39:09.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:39:10.013 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:10.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:39:10.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:39:10.141 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:10.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:39:10.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:39:10.270 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:10.277 INFO:tasks.workunit.client.0.smithi130.stdout:reading divergent objects 2020-03-11T22:39:10.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:39:10.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:10.596 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:10.597 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:38:50.900443+0000 44'102 47:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:36:34.439926+0000 0'0 2020-03-11T22:36:34.439926+0000 0 2020-03-11T22:39:10.597 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:39:10.597 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:10.598 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:39:10.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:39:10.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:39:10.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:10.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:39:10.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:10.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:39:10.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:10.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:39:10.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:10.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:39:10.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:39:10.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:10.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:39:10.748 INFO:tasks.workunit.client.0.smithi130.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:39:10.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:10.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:39:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:39:10.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:39:10.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:10.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:10.782 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:36:57.527334+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:10.782 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:36:57.527334+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:10.782 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:10.782 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2020-03-11T22:39:10.782 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4654.0:1 2020-03-11T22:37:51.041725+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:39:10.782 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2020-03-11T22:36:57.508671+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.783 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: keeping 28'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2020-03-11T22:36:57.508671+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.783 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:10.783 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'102 last_divergent_update: 28'102 2020-03-11T22:39:10.783 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:39:10.783 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:39:10.783 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'104 (28'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:36:57.525403+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:10.783 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: keeping 28'104 (28'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:36:57.525403+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:10.784 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:39:10.784 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:39:10.784 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:39:10.784 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:39:10.784 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2020-03-11T22:39:10.784 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'104 (28'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:36:57.525403+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:39:10.784 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:39:10.785 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2020-03-11T22:39:10.785 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'105 (28'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-03-11T22:36:57.525368+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.785 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: keeping 28'105 (28'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-03-11T22:36:57.525368+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.785 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:10.785 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'105 last_divergent_update: 28'105 2020-03-11T22:39:10.785 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:39:10.785 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:39:10.786 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2020-03-11T22:39:10.786 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'105 (28'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-03-11T22:36:57.525368+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.786 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:39:10.786 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2020-03-11T22:39:10.786 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:36:57.523378+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.786 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.864+0000 7f57fd61d700 20 _merge_object_divergent_entries: keeping 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:36:57.523378+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.786 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:10.787 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:39:10.787 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:39:10.788 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:39:10.788 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2020-03-11T22:39:10.788 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:36:57.523378+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.788 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:39:10.788 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2020-03-11T22:39:10.788 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:36:57.506082+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.788 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 20 _merge_object_divergent_entries: keeping 28'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:36:57.506082+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:39:10.789 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:10.789 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:39:10.789 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:39:10.789 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:33.865+0000 7f57fd61d700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:39:10.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:10.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:10.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:39:10.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17424.out 2020-03-11T22:39:10.791 INFO:tasks.workunit.client.0.smithi130.stdout:success 2020-03-11T22:39:10.791 INFO:tasks.workunit.client.0.smithi130.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17424.out': No such file or directory 2020-03-11T22:39:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:39:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:39:10.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:39:11.251 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' does not exist 2020-03-11T22:39:11.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:39:11.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:11.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:11.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:11.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:11.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:11.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:11.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:39:11.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:39:11.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:39:11.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:39:11.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:11.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:11.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:11.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:11.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:11.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:11.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:39:11.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:39:11.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:39:11.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:39:11.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:39:11.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:39:11.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:11.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:39:11.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:39:11.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:11.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:39:11.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:39:11.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:39:11.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:39:11.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:11.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:11.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:39:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:39:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:39:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:39:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:39:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:39:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:39:11.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:39:11.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:39:11.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:39:11.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:11.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:11.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:11.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:11.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:11.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:11.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:39:11.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:39:11.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:39:11.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:39:11.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:39:11.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:39:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:39:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:39:11.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:11.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:39:11.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:39:11.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:39:11.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:39:11.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:11.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:11.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:39:11.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:39:11.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:39:11.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:39:11.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:39:11.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:11.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:11.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17424 2020-03-11T22:39:11.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:39:11.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:39:11.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:39:11.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:39:11.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:39:11.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:39:11.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:39:11.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:39:11.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:39:11.407 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:39:11.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:39:11.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:39:11.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:39:11.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:39:11.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:39:11.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:11.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:11.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:11.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:11.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:11.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:11.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:39:11.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:39:11.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:39:11.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:39:11.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:39:11.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:39:11.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:39:11.903 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:39:11.903 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:39:11.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:39:11.904 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:11.904 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:11.904 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:11.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:39:11.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:39:11.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get fsid 2020-03-11T22:39:11.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:39:11.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:39:11.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:39:11.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:39:11.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:39:11.991 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:39:11.991 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:39:11.991 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get mon_host 2020-03-11T22:39:12.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:39:12.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:39:12.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:39:12.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:39:12.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:39:12.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:39:12.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:39:12.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:39:12.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:12.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:12.550 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:12.550 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:12.551 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:12.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:12.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:12.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:12.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:12.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:39:12.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:12.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:12.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:39:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:39:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:12.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:12.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:12.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:12.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:12.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:12.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:12.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:12.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:12.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:12.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:12.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:12.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:12.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:12.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:12.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:12.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:12.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:39:12.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:12.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=dad61a67-7946-4e56-97b6-52baacd3a88d 2020-03-11T22:39:12.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 dad61a67-7946-4e56-97b6-52baacd3a88d' 2020-03-11T22:39:12.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:12.586 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 dad61a67-7946-4e56-97b6-52baacd3a88d 2020-03-11T22:39:12.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCQaGleWjKRIxAA3siB/Fw+HbUpPve7Eh0ySQ== 2020-03-11T22:39:12.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCQaGleWjKRIxAA3siB/Fw+HbUpPve7Eh0ySQ=="}' 2020-03-11T22:39:12.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new dad61a67-7946-4e56-97b6-52baacd3a88d -i td/divergent-priors/0/new.json 2020-03-11T22:39:12.941 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:39:12.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:39:12.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCQaGleWjKRIxAA3siB/Fw+HbUpPve7Eh0ySQ== --osd-uuid dad61a67-7946-4e56-97b6-52baacd3a88d 2020-03-11T22:39:12.980 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:12.976+0000 7fc011a4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:12.979+0000 7fc011a4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:12.980+0000 7fc011a4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:12.984 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:12.981+0000 7fc011a4cec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:12.984 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:12.981+0000 7fc011a4cec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:12.984 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:12.981+0000 7fc011a4cec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:13.049 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:13.046+0000 7fc011a4cec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:39:16.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:39:16.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:16.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:39:16.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:16.009 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:39:16.455 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:39:16.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:39:16.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:16.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:16.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:16.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:16.482 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:16.478+0000 7fdbb582aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:16.483 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:16.480+0000 7fdbb582aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:16.484 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:16.481+0000 7fdbb582aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:16.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:39:16.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:16.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:16.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:16.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:16.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:16.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:16.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:16.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:16.803 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:39:17.000 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:16.997+0000 7fdbb582aec0 -1 Falling back to public interface 2020-03-11T22:39:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:17.774 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:17.771+0000 7fdbb582aec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:39:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:18.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:18.176 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:39:18.528 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34529,v1:127.0.0.1:6803/34529] [v2:127.0.0.1:6804/34529,v1:127.0.0.1:6805/34529] exists,up dad61a67-7946-4e56-97b6-52baacd3a88d 2020-03-11T22:39:18.528 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 53637eac-f654-4841-82d1-b00b8684e01b 2020-03-11T22:39:18.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:18.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:18.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:39:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:39:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:18.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:18.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:18.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:18.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:18.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:18.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:18.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:18.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:18.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:18.534 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:18.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:18.537 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:18.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:18.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:18.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:18.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:18.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:18.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:18.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:18.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:18.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:18.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:18.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:18.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:18.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:18.539 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:18.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=53637eac-f654-4841-82d1-b00b8684e01b 2020-03-11T22:39:18.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 53637eac-f654-4841-82d1-b00b8684e01b' 2020-03-11T22:39:18.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:18.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCWaGleibBkIBAARImhhYJxsfFmXAtv9ve2Jg== 2020-03-11T22:39:18.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCWaGleibBkIBAARImhhYJxsfFmXAtv9ve2Jg=="}' 2020-03-11T22:39:18.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 53637eac-f654-4841-82d1-b00b8684e01b -i td/divergent-priors/1/new.json 2020-03-11T22:39:19.076 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:39:19.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:39:19.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCWaGleibBkIBAARImhhYJxsfFmXAtv9ve2Jg== --osd-uuid 53637eac-f654-4841-82d1-b00b8684e01b 2020-03-11T22:39:19.115 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:19.111+0000 7fd817567ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:19.116 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:19.113+0000 7fd817567ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:19.118 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:19.114+0000 7fd817567ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:19.118 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:19.115+0000 7fd817567ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:19.119 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:19.115+0000 7fd817567ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:19.119 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:19.115+0000 7fd817567ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:19.179 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:19.175+0000 7fd817567ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:39:22.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:39:22.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:22.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:39:22.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:22.100 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:39:22.556 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:39:22.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:39:22.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:22.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:22.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:22.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:22.583 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:22.580+0000 7f999c566ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.584 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:22.581+0000 7f999c566ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.586 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:22.583+0000 7f999c566ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:39:22.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:22.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:39:22.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:22.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:22.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:22.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:22.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:22.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:22.901 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:39:23.102 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:23.098+0000 7f999c566ec0 -1 Falling back to public interface 2020-03-11T22:39:23.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:23.988 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:23.985+0000 7f999c566ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:39:24.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:24.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:24.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:24.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:24.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:24.246 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:39:24.641 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34884,v1:127.0.0.1:6811/34884] [v2:127.0.0.1:6812/34884,v1:127.0.0.1:6813/34884] exists,up 53637eac-f654-4841-82d1-b00b8684e01b 2020-03-11T22:39:24.641 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 41240656-1515-44bf-9b8c-f85a0a4cd9c5 2020-03-11T22:39:24.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:24.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:24.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:39:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:39:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:39:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:39:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:39:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:39:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:24.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:39:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:39:24.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:24.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=41240656-1515-44bf-9b8c-f85a0a4cd9c5 2020-03-11T22:39:24.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 41240656-1515-44bf-9b8c-f85a0a4cd9c5' 2020-03-11T22:39:24.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:24.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCcaGleLlYeJxAAlpM3qpKQ/3ZZ34xjn4OzvQ== 2020-03-11T22:39:24.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCcaGleLlYeJxAAlpM3qpKQ/3ZZ34xjn4OzvQ=="}' 2020-03-11T22:39:24.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 41240656-1515-44bf-9b8c-f85a0a4cd9c5 -i td/divergent-priors/2/new.json 2020-03-11T22:39:25.144 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:39:25.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:39:25.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCcaGleLlYeJxAAlpM3qpKQ/3ZZ34xjn4OzvQ== --osd-uuid 41240656-1515-44bf-9b8c-f85a0a4cd9c5 2020-03-11T22:39:25.183 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:25.180+0000 7f27d4e18ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:25.185 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:25.182+0000 7f27d4e18ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:25.187 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:25.184+0000 7f27d4e18ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:25.188 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:25.184+0000 7f27d4e18ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:25.188 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:25.184+0000 7f27d4e18ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:25.188 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:25.184+0000 7f27d4e18ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:25.253 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:25.250+0000 7f27d4e18ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:39:28.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:39:28.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:28.320 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:39:28.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:39:28.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:28.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:39:28.769 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:39:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:28.795 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:28.792+0000 7f1924de4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:28.796 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:28.793+0000 7f1924de4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:28.798 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:28.795+0000 7f1924de4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:29.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:39:29.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:29.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:39:29.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:29.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:29.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:29.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:29.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:29.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:29.119 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:39:29.313 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:29.309+0000 7f1924de4ec0 -1 Falling back to public interface 2020-03-11T22:39:29.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:30.085 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:39:30.082+0000 7f1924de4ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:39:30.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:30.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:30.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:30.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:30.463 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:39:30.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:30.834 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35239,v1:127.0.0.1:6819/35239] [v2:127.0.0.1:6820/35239,v1:127.0.0.1:6821/35239] exists,up 41240656-1515-44bf-9b8c-f85a0a4cd9c5 2020-03-11T22:39:30.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:30.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:30.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:30.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:39:31.561 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2020-03-11T22:39:31.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:39:32.221 INFO:tasks.workunit.client.0.smithi130.stderr:noin is set 2020-03-11T22:39:32.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:39:32.931 INFO:tasks.workunit.client.0.smithi130.stderr:nodown is set 2020-03-11T22:39:32.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:39:32.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:39:33.402 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2020-03-11T22:39:33.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:39:34.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:39:34.968 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 3 2020-03-11T22:39:34.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:39:35.610 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 min_size to 2 2020-03-11T22:39:35.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:39:36.354 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:39:36.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:39:36.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:36.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:36.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:36.708 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:39:36.708 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:39:36.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:36.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:36.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:36.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:39:36.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:39:36.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:36.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:36.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:39:36.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:39:36.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:36.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:37.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:39:37.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:39:37.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:37.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:39:37.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:37.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:37.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:39:37.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:37.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:39:37.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:39:37.115 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:39:37.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:37.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:39:37.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:38.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:38.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:38.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:39:38.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:38.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:38.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:38.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:38.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:38.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:38.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:39:38.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:39:38.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:38.818 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:39:39.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:39:39.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:39.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:39:39.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:39.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:39.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:39:39.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:39.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:39:39.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:39:39.199 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:39:39.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:39.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:39:39.565 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:39.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:39.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:39.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:39.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:39.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:39.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:39.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:39.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:39.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:40.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:39:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:39:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:40.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:40.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:39:40.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:39:40.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:40.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:40.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:39:40.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-03-11T22:39:40.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:40.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:40.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:39:40.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446' 2020-03-11T22:39:40.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:40.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:39:40.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:40.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:40.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:39:40.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:40.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:39:40.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:39:40.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:40.398 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:39:40.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:39:40.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:41.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:41.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:42.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:39:42.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:42.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:39:42.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:42.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:42.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:39:42.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:42.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:39:42.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:39:42.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:42.095 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:39:42.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672967 2020-03-11T22:39:42.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:42.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:39:42.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:42.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:42.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:39:42.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:42.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:39:42.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:39:42.443 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:39:42.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:42.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:39:42.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:42.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:42.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:43.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:43.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:43.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:43.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:43.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:43.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:43.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:43.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:43.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:43.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:44.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:39:44.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:39:44.359 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:39:44.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:39:44.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:39:44.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:44.370 INFO:tasks.workunit.client.0.smithi130.stdout:primary and soon to be divergent is 1 2020-03-11T22:39:44.688 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:44.689 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:39:34.103939+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:39:44.689 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:39:44.689 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:44.690 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:39:44.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:39:44.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:44.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:39:44.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:39:44.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:44.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:39:44.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:39:44.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:39:44.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:39:44.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:39:44.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:39:44.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:39:44.700 INFO:tasks.workunit.client.0.smithi130.stdout:writing initial objects 2020-03-11T22:39:44.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:44.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:39:44.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:44.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:39:44.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:44.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:39:44.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:44.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:39:44.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:44.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:39:44.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:44.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:39:45.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:39:45.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:39:45.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:39:45.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:39:45.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:39:45.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:39:45.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:39:45.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:39:45.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:39:45.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:39:45.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:39:45.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:39:45.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:39:45.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:45.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:39:46.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:39:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:39:46.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:39:46.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:39:46.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:39:46.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:39:46.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:39:46.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:39:46.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:39:46.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:39:46.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:39:46.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:39:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:39:46.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:39:46.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:39:46.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:39:46.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:39:46.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:46.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:39:47.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:39:47.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:39:47.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:39:47.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:39:47.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:39:47.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:39:47.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:39:47.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:39:47.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:39:47.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:39:47.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:39:47.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:39:47.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:39:47.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:47.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:39:48.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:39:48.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:39:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:39:48.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:39:48.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:39:48.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:39:48.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:39:48.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:39:48.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:39:48.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:39:48.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:39:48.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:39:48.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:39:48.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:39:48.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:39:48.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:39:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:39:49.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:39:49.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:39:49.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:39:49.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:39:49.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:39:49.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:39:49.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:39:49.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:39:49.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:39:49.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:39:49.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:39:49.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:39:49.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:39:49.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:39:49.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:39:49.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:39:49.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:39:50.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:39:50.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:39:50.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:39:50.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:39:50.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:39:50.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:39:50.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:39:50.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:39:50.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:39:50.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:39:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:50.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:50.768 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:50.769 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:50.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:50.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:50.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:50.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:50.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:50.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:50.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:50.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:51.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:51.176 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:39:51.176 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:39:51.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:51.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:51.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:51.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:39:51.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:39:51.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:51.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:51.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:39:51.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:39:51.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:51.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:51.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:39:51.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:39:51.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:51.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:39:51.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:51.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:51.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:39:51.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:51.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:39:51.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:39:51.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:51.569 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:39:51.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:39:51.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:52.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:52.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:53.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:39:53.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:53.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:53.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:39:53.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:53.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:39:53.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:39:53.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:53.264 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:39:53.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:39:53.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:53.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:39:53.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:53.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:53.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:39:53.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:53.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:39:53.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:39:53.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:53.760 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:39:53.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:39:53.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:53.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:53.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:54.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:54.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:54.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:54.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:54.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:54.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:54.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:54.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:54.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:54.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:54.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:54.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:55.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:55.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:55.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:55.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:39:55.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:39:55.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:39:55.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:39:55.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:39:55.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:39:55.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:39:55.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:55.554 INFO:tasks.workunit.client.0.smithi130.stdout:blackholing osds 0 2 2020-03-11T22:39:55.878 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:55.878 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:39:34.103939+0000 30'100 30:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:39:55.878 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:39:55.878 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:55.880 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:39:55.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:55.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:39:55.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:39:55.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:39:55.889 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:55.889 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:55.889 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:55.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:39:55.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:39:55.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:39:55.975 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:39:55.975 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:39:55.975 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:39:55.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:55.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:39:55.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:39:55.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:39:55.984 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:55.984 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:55.984 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:39:55.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:39:55.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:39:55.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:39:56.070 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:39:56.070 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:39:56.070 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:39:56.077 INFO:tasks.workunit.client.0.smithi130.stdout:create new divergent objects 2020-03-11T22:39:56.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:39:56.078 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:39:56.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:56.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:39:56.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:39:56.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:39:56.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:39:56.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:39:56.080 INFO:tasks.workunit.client.0.smithi130.stdout:writing divergent objects 2020-03-11T22:39:56.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:39:56.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:39:56.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:39:56.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:39:56.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:39:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:39:56.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:39:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:39:56.082 INFO:tasks.workunit.client.0.smithi130.stdout:remove divergent objects 2020-03-11T22:39:56.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:39:56.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:39:56.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:39:56.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:39:56.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:39:56.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:39:56.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:40:06.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:40:06.090 INFO:tasks.workunit.client.0.smithi130.stderr:rados: no process found 2020-03-11T22:40:06.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:40:06.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:06.091 INFO:tasks.workunit.client.0.smithi130.stdout:killing all the osds 2020-03-11T22:40:06.418 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:06.418 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:39:34.103939+0000 30'106 30:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:40:06.418 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:06.419 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:40:06.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:40:06.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:06.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:06.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:06.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:06.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:06.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:06.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:40:06.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:40:07.012 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2020-03-11T22:40:07.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:40:07.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:40:07.600 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already down. 2020-03-11T22:40:07.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:40:07.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:40:08.086 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already down. 2020-03-11T22:40:08.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:08.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:40:08.562 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2020-03-11T22:40:08.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:08.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:40:09.036 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2020-03-11T22:40:09.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:40:09.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:09.045 INFO:tasks.workunit.client.0.smithi130.stdout:bringing up non_divergent 0 2 2020-03-11T22:40:09.363 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:09.363 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:39:34.103939+0000 30'106 30:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:09.363 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:40:09.364 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:09.365 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:40:09.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:09.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:40:09.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:09.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:09.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:40:09.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:09.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:40:09.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:09.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:09.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:09.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:09.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:40:09.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:40:09.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:09.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:09.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:09.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:09.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:09.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:09.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:09.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:09.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:40:09.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:40:09.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:40:09.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:09.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:09.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:09.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:09.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:09.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:09.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:09.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:09.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:40:09.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:40:09.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:09.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:40:09.382 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:40:09.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:40:09.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:09.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:09.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:09.402 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:09.399+0000 7fcf4223cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:09.404 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:09.400+0000 7fcf4223cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:09.406 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:09.403+0000 7fcf4223cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:09.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:40:09.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:09.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:09.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:09.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:09.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:09.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:09.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:09.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:09.725 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:40:09.922 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:09.918+0000 7fcf4223cec0 -1 Falling back to public interface 2020-03-11T22:40:10.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:10.800 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:10.797+0000 7fcf4223cec0 -1 osd.0 30 log_to_monitors {default=true} 2020-03-11T22:40:11.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:11.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:11.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:11.068 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:40:11.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:11.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:11.425 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39310,v1:127.0.0.1:6803/39310] [v2:127.0.0.1:6804/39310,v1:127.0.0.1:6805/39310] exists,up dad61a67-7946-4e56-97b6-52baacd3a88d 2020-03-11T22:40:11.425 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:40:11.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:11.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:11.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:11.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:11.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:40:11.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:11.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:11.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:40:11.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:11.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:40:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:40:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:40:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:11.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:11.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:11.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:11.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:11.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:11.428 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:11.428 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:11.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:40:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:40:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:11.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:40:11.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:40:11.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:11.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:11.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:11.451 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:11.447+0000 7f0b14df1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:11.452 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:11.449+0000 7f0b14df1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:11.454 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:11.450+0000 7f0b14df1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:11.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:40:11.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:11.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:11.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:11.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:11.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:11.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:11.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:11.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:11.801 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:40:11.972 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:11.968+0000 7f0b14df1ec0 -1 Falling back to public interface 2020-03-11T22:40:12.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:13.074 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:40:13.070+0000 7f0b14df1ec0 -1 osd.2 30 log_to_monitors {default=true} 2020-03-11T22:40:13.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:13.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:13.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:13.150 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:40:13.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:13.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:13.524 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,30) [v2:127.0.0.1:6810/39504,v1:127.0.0.1:6811/39504] [v2:127.0.0.1:6812/39504,v1:127.0.0.1:6813/39504] exists,up 41240656-1515-44bf-9b8c-f85a0a4cd9c5 2020-03-11T22:40:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:13.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:40:14.037 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already in. 2020-03-11T22:40:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:14.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:40:14.702 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2020-03-11T22:40:14.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:14.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:40:14.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:14.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:14.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:14.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:14.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:14.712 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:14.712 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:14.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:14.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:14.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:14.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:15.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:15.110 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:40:15.110 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:40:15.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:15.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:15.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:15.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:40:15.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659' 2020-03-11T22:40:15.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:15.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:15.375 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:15.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:15.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1-' 2020-03-11T22:40:15.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:15.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:15.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789955 2020-03-11T22:40:15.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1- 2-158913789955' 2020-03-11T22:40:15.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:15.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822659 2020-03-11T22:40:15.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:15.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:15.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822659 2020-03-11T22:40:15.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:15.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:40:15.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2020-03-11T22:40:15.514 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 154618822659 2020-03-11T22:40:15.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:15.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 154618822659 2020-03-11T22:40:15.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:16.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:16.862 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:17.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822659 -lt 154618822659 2020-03-11T22:40:17.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:17.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:17.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:17.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:17.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:17.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:17.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:17.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:17.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:17.213 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 2020-03-11T22:40:17.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:40:17.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:40:17.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:17.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789955 2020-03-11T22:40:17.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:17.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:17.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789955 2020-03-11T22:40:17.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:17.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789955 2020-03-11T22:40:17.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2020-03-11T22:40:17.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:17.583 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 158913789955 2020-03-11T22:40:17.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789955 -lt 158913789955 2020-03-11T22:40:17.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:17.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:17.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:18.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:40:18.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:18.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:18.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:18.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:18.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:18.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:18.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:18.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:18.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:18.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:18.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:19.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:40:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:40:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:40:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:40:19.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:19.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:19.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:19.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:19.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:19.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:19.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:19.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:19.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:19.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:19.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:19.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:20.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:20.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:20.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:20.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:20.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:20.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:20.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:20.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:20.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:20.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:20.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:20.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:40:20.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:20.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:40:20.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:20.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:20.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:21.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:21.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:21.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:21.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:21.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:21.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:21.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:21.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:21.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:21.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:21.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:21.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:21.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:21.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:21.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:21.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:40:21.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:21.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:40:22.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:22.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:22.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:22.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:22.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:22.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:22.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:22.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:22.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:22.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:22.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:22.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:23.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:23.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:23.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:23.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:23.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:23.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:23.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:23.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:23.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:23.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:23.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:23.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:40:23.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:23.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:40:24.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:24.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:24.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:24.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:24.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:24.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:24.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:24.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:24.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:24.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:24.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:24.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:25.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:25.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:25.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:25.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:25.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:25.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:25.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:25.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:25.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:25.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:25.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:25.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:40:25.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:25.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:40:27.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:27.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:27.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:27.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:27.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:27.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:27.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:27.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:27.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:27.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:27.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:27.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:28.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:28.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:28.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:28.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:28.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:28.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:28.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:28.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:28.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:28.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:28.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:28.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:40:28.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:28.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:40:31.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:31.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:31.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:31.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:31.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:31.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:31.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:31.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:32.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:32.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:32.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:32.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:32.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:32.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:32.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:32.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:32.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:32.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:32.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:32.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:32.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:32.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:32.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:32.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:40:32.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:32.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:40:39.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:39.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:39.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:39.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:39.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:39.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:39.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:39.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:39.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:39.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:40.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:40.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:40.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:40.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:40.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:40.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:40.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:40.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:40.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:40.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:40.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:40.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:40:40.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:40.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:40:47.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:47.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:47.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:47.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:47.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:47.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:47.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:47.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:48.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:48.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:48.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:48.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:48.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:48.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:48.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:48.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:48.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:48.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:48.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:48.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:48.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:49.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:49.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:49.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:40:49.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:40:49.577 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:40:49.577 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_fingerprint": "0c682c5e-762b-4242-a76c-209ee2638d46", 2020-03-11T22:40:49.577 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:40:49.577 INFO:tasks.workunit.client.0.smithi130.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:40:49.577 INFO:tasks.workunit.client.0.smithi130.stdout: "timestamp": "2020-03-11T22:40:49.573902+0000", 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi130.stdout: "tag": "", 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi130.stdout: "health": { 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "HEALTH_WARN", 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi130.stdout: "checks": { 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi130.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:40:49.579 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:40:49.579 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 17 2020-03-11T22:40:49.579 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.579 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [], 2020-03-11T22:40:49.579 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:40:49.579 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: "OSD_DOWN": { 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 osds down", 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:40:49.580 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "osd.1 (root=default,host=smithi130) is down" 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: "PG_DEGRADED": { 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:49.581 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:40:49.582 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:40:49.582 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 32 2020-03-11T22:40:49.582 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.582 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:40:49.582 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.582 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.582 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.583 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.583 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.583 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.583 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.583 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.583 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.584 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.584 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.584 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.584 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.584 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.584 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.584 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.585 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.585 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.585 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.585 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.585 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.585 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.586 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.586 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.586 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.586 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.586 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.587 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.587 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.587 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.587 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.587 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.588 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.588 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.588 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.588 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.588 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.589 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.589 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.589 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.589 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.590 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.590 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.590 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.590 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.591 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.591 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.591 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.591 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.592 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.592 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.592 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.592 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.592 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.593 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.593 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.593 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.593 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.594 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.594 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.594 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.594 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.594 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.595 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.595 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.595 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.595 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.596 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.596 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.596 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.596 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.596 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.597 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.597 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.597 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.597 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.598 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.598 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.598 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.598 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.599 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.599 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.599 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.599 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.599 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.600 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.600 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.600 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.600 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.601 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.601 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.601 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.601 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.601 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.602 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:49.602 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.602 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.602 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:40:49.603 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.603 INFO:tasks.workunit.client.0.smithi130.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:40:49.603 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:49.603 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:40:49.603 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:40:49.604 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:40:49.604 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.604 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:40:49.604 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.604 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:40:49.605 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.605 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.605 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:40:49.605 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.605 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.606 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:40:49.606 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.606 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.606 INFO:tasks.workunit.client.0.smithi130.stdout: "mutes": [] 2020-03-11T22:40:49.607 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.607 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_first_committed": 1, 2020-03-11T22:40:49.607 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_last_committed": 1, 2020-03-11T22:40:49.607 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap": { 2020-03-11T22:40:49.607 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:40:49.608 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:40:49.608 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:39:11.426694Z", 2020-03-11T22:40:49.608 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:39:11.426694Z", 2020-03-11T22:40:49.608 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release": 15, 2020-03-11T22:40:49.609 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:40:49.609 INFO:tasks.workunit.client.0.smithi130.stdout: "features": { 2020-03-11T22:40:49.609 INFO:tasks.workunit.client.0.smithi130.stdout: "persistent": [ 2020-03-11T22:40:49.609 INFO:tasks.workunit.client.0.smithi130.stdout: "kraken", 2020-03-11T22:40:49.609 INFO:tasks.workunit.client.0.smithi130.stdout: "luminous", 2020-03-11T22:40:49.610 INFO:tasks.workunit.client.0.smithi130.stdout: "mimic", 2020-03-11T22:40:49.610 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap-prune", 2020-03-11T22:40:49.610 INFO:tasks.workunit.client.0.smithi130.stdout: "nautilus", 2020-03-11T22:40:49.610 INFO:tasks.workunit.client.0.smithi130.stdout: "octopus" 2020-03-11T22:40:49.611 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.611 INFO:tasks.workunit.client.0.smithi130.stdout: "optional": [] 2020-03-11T22:40:49.611 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.611 INFO:tasks.workunit.client.0.smithi130.stdout: "mons": [ 2020-03-11T22:40:49.611 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.612 INFO:tasks.workunit.client.0.smithi130.stdout: "rank": 0, 2020-03-11T22:40:49.612 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "a", 2020-03-11T22:40:49.612 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:40:49.612 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.613 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.613 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.613 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:40:49.613 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 0 2020-03-11T22:40:49.613 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.614 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.614 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.614 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:40:49.614 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:40:49.615 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 0, 2020-03-11T22:40:49.615 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 0 2020-03-11T22:40:49.615 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.615 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.615 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.616 INFO:tasks.workunit.client.0.smithi130.stdout: "quorum": [ 2020-03-11T22:40:49.616 INFO:tasks.workunit.client.0.smithi130.stdout: 0 2020-03-11T22:40:49.616 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.616 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap": { 2020-03-11T22:40:49.617 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 43, 2020-03-11T22:40:49.617 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:40:49.617 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:39:12.120944+0000", 2020-03-11T22:40:49.617 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:40:18.133240+0000", 2020-03-11T22:40:49.617 INFO:tasks.workunit.client.0.smithi130.stdout: "last_up_change": "2020-03-11T22:40:13.149407+0000", 2020-03-11T22:40:49.617 INFO:tasks.workunit.client.0.smithi130.stdout: "last_in_change": "2020-03-11T22:40:14.362860+0000", 2020-03-11T22:40:49.618 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:40:49.618 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_num": 5800384, 2020-03-11T22:40:49.618 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_set": [ 2020-03-11T22:40:49.618 INFO:tasks.workunit.client.0.smithi130.stdout: "nodown", 2020-03-11T22:40:49.619 INFO:tasks.workunit.client.0.smithi130.stdout: "noin", 2020-03-11T22:40:49.619 INFO:tasks.workunit.client.0.smithi130.stdout: "noout", 2020-03-11T22:40:49.619 INFO:tasks.workunit.client.0.smithi130.stdout: "pglog_hardlimit", 2020-03-11T22:40:49.619 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snapdirs", 2020-03-11T22:40:49.619 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_deletes", 2020-03-11T22:40:49.620 INFO:tasks.workunit.client.0.smithi130.stdout: "sortbitwise" 2020-03-11T22:40:49.620 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.620 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_version": 7, 2020-03-11T22:40:49.621 INFO:tasks.workunit.client.0.smithi130.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:40:49.621 INFO:tasks.workunit.client.0.smithi130.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:40:49.621 INFO:tasks.workunit.client.0.smithi130.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:40:49.621 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_snapshot": "", 2020-03-11T22:40:49.622 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_max": 1, 2020-03-11T22:40:49.622 INFO:tasks.workunit.client.0.smithi130.stdout: "max_osd": 3, 2020-03-11T22:40:49.622 INFO:tasks.workunit.client.0.smithi130.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:40:49.622 INFO:tasks.workunit.client.0.smithi130.stdout: "min_compat_client": "jewel", 2020-03-11T22:40:49.623 INFO:tasks.workunit.client.0.smithi130.stdout: "require_osd_release": "octopus", 2020-03-11T22:40:49.623 INFO:tasks.workunit.client.0.smithi130.stdout: "pools": [ 2020-03-11T22:40:49.623 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.624 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:40:49.624 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_name": "test", 2020-03-11T22:40:49.624 INFO:tasks.workunit.client.0.smithi130.stdout: "create_time": "2020-03-11T22:39:33.194196+0000", 2020-03-11T22:40:49.624 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": 1, 2020-03-11T22:40:49.624 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_names": "hashpspool", 2020-03-11T22:40:49.624 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:40:49.625 INFO:tasks.workunit.client.0.smithi130.stdout: "size": 3, 2020-03-11T22:40:49.625 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 2, 2020-03-11T22:40:49.625 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_rule": 0, 2020-03-11T22:40:49.625 INFO:tasks.workunit.client.0.smithi130.stdout: "object_hash": 2, 2020-03-11T22:40:49.625 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:40:49.626 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num": 32, 2020-03-11T22:40:49.626 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num": 1, 2020-03-11T22:40:49.626 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num_target": 32, 2020-03-11T22:40:49.626 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_target": 32, 2020-03-11T22:40:49.626 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_pending": 32, 2020-03-11T22:40:49.626 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pg_merge_meta": { 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi130.stdout: "source_pgid": "0.0", 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi130.stdout: "ready_epoch": 0, 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 0, 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 0, 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi130.stdout: "source_version": "0'0", 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi130.stdout: "target_version": "0'0" 2020-03-11T22:40:49.628 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.628 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "42", 2020-03-11T22:40:49.628 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend": "0", 2020-03-11T22:40:49.628 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:40:49.628 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_preluminous": "42", 2020-03-11T22:40:49.629 INFO:tasks.workunit.client.0.smithi130.stdout: "auid": 0, 2020-03-11T22:40:49.629 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:40:49.629 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_seq": 0, 2020-03-11T22:40:49.629 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_epoch": 0, 2020-03-11T22:40:49.629 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_snaps": [], 2020-03-11T22:40:49.630 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps": "[]", 2020-03-11T22:40:49.630 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_bytes": 0, 2020-03-11T22:40:49.630 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_objects": 0, 2020-03-11T22:40:49.630 INFO:tasks.workunit.client.0.smithi130.stdout: "tiers": [], 2020-03-11T22:40:49.630 INFO:tasks.workunit.client.0.smithi130.stdout: "tier_of": -1, 2020-03-11T22:40:49.630 INFO:tasks.workunit.client.0.smithi130.stdout: "read_tier": -1, 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "write_tier": -1, 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_mode": "none", 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_bytes": 0, 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_objects": 0, 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:40:49.631 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_flush_age": 0, 2020-03-11T22:40:49.632 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_evict_age": 0, 2020-03-11T22:40:49.632 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profile": "", 2020-03-11T22:40:49.632 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_params": { 2020-03-11T22:40:49.632 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "none" 2020-03-11T22:40:49.632 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.632 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_period": 0, 2020-03-11T22:40:49.632 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_count": 0, 2020-03-11T22:40:49.633 INFO:tasks.workunit.client.0.smithi130.stdout: "use_gmt_hitset": true, 2020-03-11T22:40:49.633 INFO:tasks.workunit.client.0.smithi130.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:40:49.633 INFO:tasks.workunit.client.0.smithi130.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:40:49.633 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:40:49.633 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:40:49.633 INFO:tasks.workunit.client.0.smithi130.stdout: "grade_table": [], 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: "stripe_width": 0, 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: "expected_num_objects": 0, 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: "fast_read": false, 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: "options": {}, 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: "application_metadata": {} 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.634 INFO:tasks.workunit.client.0.smithi130.stdout: "osds": [ 2020-03-11T22:40:49.635 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.635 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:40:49.635 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "dad61a67-7946-4e56-97b6-52baacd3a88d", 2020-03-11T22:40:49.636 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:40:49.636 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:40:49.637 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:40:49.637 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:40:49.637 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 5, 2020-03-11T22:40:49.637 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 30, 2020-03-11T22:40:49.637 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 36, 2020-03-11T22:40:49.637 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 42, 2020-03-11T22:40:49.638 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 31, 2020-03-11T22:40:49.638 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:40:49.638 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:40:49.638 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.638 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.638 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.639 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:40:49.640 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.641 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.641 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.641 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.641 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:40:49.641 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.641 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.641 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.642 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.642 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:49.642 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.642 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.642 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.643 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:40:49.643 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.643 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.643 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.646 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.646 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:40:49.646 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.647 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.647 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.647 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.647 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:49.647 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.648 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.648 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.648 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:40:49.648 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.648 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.648 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.649 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.649 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:40:49.649 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:40:49.649 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.649 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.649 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.649 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6803/39310", 2020-03-11T22:40:49.650 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6805/39310", 2020-03-11T22:40:49.650 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39310", 2020-03-11T22:40:49.650 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39310", 2020-03-11T22:40:49.650 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:40:49.650 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:40:49.650 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:40:49.650 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.651 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.651 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.651 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:40:49.651 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "53637eac-f654-4841-82d1-b00b8684e01b", 2020-03-11T22:40:49.651 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 0, 2020-03-11T22:40:49.651 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:40:49.651 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 0, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 0, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 10, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 22, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 32, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:40:49.652 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:40:49.653 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.653 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.653 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.653 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:49.653 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.653 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.654 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.654 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.654 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:49.654 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.654 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.654 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.654 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.655 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:40:49.655 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.655 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.655 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.655 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:49.655 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.655 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.656 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:49.657 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.657 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.657 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.657 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:49.657 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.657 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.657 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:49.658 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.659 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:49.660 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:40:49.660 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.660 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.660 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.660 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/34884", 2020-03-11T22:40:49.660 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/34884", 2020-03-11T22:40:49.660 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34884", 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34884", 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: "exists" 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:40:49.661 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "41240656-1515-44bf-9b8c-f85a0a4cd9c5", 2020-03-11T22:40:49.662 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:40:49.662 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:40:49.662 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:40:49.662 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:40:49.662 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 15, 2020-03-11T22:40:49.662 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 30, 2020-03-11T22:40:49.662 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 37, 2020-03-11T22:40:49.663 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 0, 2020-03-11T22:40:49.663 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 33, 2020-03-11T22:40:49.663 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:40:49.664 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:40:49.665 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.665 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.665 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.665 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:49.665 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.665 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.666 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.666 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.666 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:49.666 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.666 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.667 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.667 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.667 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:40:49.667 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.667 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.667 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.668 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.669 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.669 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.669 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:49.669 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.669 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.669 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.669 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.670 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.671 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.671 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:49.671 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:40:49.671 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.671 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:40:49.671 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:49.671 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.672 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.672 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.672 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:40:49.672 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:49.672 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:40:49.672 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.672 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.673 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.673 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/39504", 2020-03-11T22:40:49.673 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/39504", 2020-03-11T22:40:49.673 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39504", 2020-03-11T22:40:49.673 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39504", 2020-03-11T22:40:49.673 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:40:49.673 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:40:49.674 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:40:49.674 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.674 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.674 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.674 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_xinfo": [ 2020-03-11T22:40:49.674 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.674 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:40:49.675 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:40:06.831157+0000", 2020-03-11T22:40:49.675 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:40:49.675 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:40:49.675 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:40:49.675 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:40:49.675 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:18.723697+0000", 2020-03-11T22:40:49.675 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:40:07.386483+0000", 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:40:49.676 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:24.974450+0000", 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:40:07.920971+0000", 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:40:49.677 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:40:49.678 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:40:49.678 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:40:49.678 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:31.102932+0000", 2020-03-11T22:40:49.678 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:40:49.678 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.678 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.678 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap": [], 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap_items": [], 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_temp": [], 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_temp": [], 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "blacklist": {}, 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profiles": { 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "default": { 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:40:49.679 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "2", 2020-03-11T22:40:49.680 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:40:49.680 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van" 2020-03-11T22:40:49.680 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.680 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.680 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps_queue": [], 2020-03-11T22:40:49.680 INFO:tasks.workunit.client.0.smithi130.stdout: "new_removed_snaps": [], 2020-03-11T22:40:49.681 INFO:tasks.workunit.client.0.smithi130.stdout: "new_purged_snaps": [], 2020-03-11T22:40:49.681 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_node_flags": {}, 2020-03-11T22:40:49.681 INFO:tasks.workunit.client.0.smithi130.stdout: "device_class_flags": {} 2020-03-11T22:40:49.681 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.681 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_metadata": [ 2020-03-11T22:40:49.681 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.681 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6804/39310,v1:127.0.0.1:6805/39310]", 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:49.683 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:49.683 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:49.683 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:49.683 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:40:49.683 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:49.683 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:49.683 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:40:49.684 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:49.685 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:40:49.685 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:40:49.685 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:49.685 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:40:49.685 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6802/39310,v1:127.0.0.1:6803/39310]", 2020-03-11T22:40:49.685 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:40:49.685 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39310,v1:127.0.0.1:6809/39310]", 2020-03-11T22:40:49.686 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39310,v1:127.0.0.1:6807/39310]", 2020-03-11T22:40:49.686 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:40:49.686 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:40:49.686 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:49.686 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:49.686 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:49.686 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:40:49.687 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:49.687 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:49.687 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:40:49.687 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:40:49.687 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:49.687 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:40:49.687 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.688 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.688 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:40:49.688 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:40:49.688 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/34884,v1:127.0.0.1:6813/34884]", 2020-03-11T22:40:49.688 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:40:49.688 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:40:49.688 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:49.689 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:49.689 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:49.689 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:49.689 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:49.689 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:49.689 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:49.690 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:40:49.690 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:49.690 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:49.690 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:49.691 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:49.691 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:40:49.691 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:49.691 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:49.691 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:40:49.691 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:40:49.692 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:40:49.692 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:49.692 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:40:49.692 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:40:49.692 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:49.692 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:40:49.692 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/34884,v1:127.0.0.1:6811/34884]", 2020-03-11T22:40:49.693 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:40:49.693 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34884,v1:127.0.0.1:6817/34884]", 2020-03-11T22:40:49.693 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34884,v1:127.0.0.1:6815/34884]", 2020-03-11T22:40:49.693 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:40:49.693 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:40:49.693 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:49.693 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:49.694 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:49.694 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:40:49.694 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:49.694 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:49.694 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:40:49.694 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:40:49.695 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:49.695 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:40:49.695 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.695 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.695 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:40:49.695 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:40:49.695 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/39504,v1:127.0.0.1:6813/39504]", 2020-03-11T22:40:49.696 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:40:49.696 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:40:49.696 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:49.696 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:49.696 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:49.696 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:49.697 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:49.697 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:49.697 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:49.697 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:40:49.697 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:49.697 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:40:49.698 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:40:49.699 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:49.699 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:40:49.699 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:40:49.699 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:49.699 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:40:49.699 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/39504,v1:127.0.0.1:6811/39504]", 2020-03-11T22:40:49.699 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39504,v1:127.0.0.1:6817/39504]", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39504,v1:127.0.0.1:6815/39504]", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:49.700 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:40:49.701 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:49.701 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:49.701 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:40:49.701 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:40:49.701 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:49.701 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:40:49.701 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_first_committed": 1, 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_last_committed": 43, 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: "crushmap": { 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": [ 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:40:49.702 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.0", 2020-03-11T22:40:49.703 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:40:49.703 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.703 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.703 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:40:49.703 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.1", 2020-03-11T22:40:49.703 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:40:49.703 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.704 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.704 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:40:49.704 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.2", 2020-03-11T22:40:49.704 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:40:49.704 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.704 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.704 INFO:tasks.workunit.client.0.smithi130.stdout: "types": [ 2020-03-11T22:40:49.705 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.705 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 0, 2020-03-11T22:40:49.705 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd" 2020-03-11T22:40:49.705 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.705 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.705 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "host" 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 2, 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "chassis" 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.706 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 3, 2020-03-11T22:40:49.707 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "rack" 2020-03-11T22:40:49.707 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.707 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.707 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 4, 2020-03-11T22:40:49.707 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "row" 2020-03-11T22:40:49.707 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.707 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.708 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 5, 2020-03-11T22:40:49.708 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pdu" 2020-03-11T22:40:49.708 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.708 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.708 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 6, 2020-03-11T22:40:49.708 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pod" 2020-03-11T22:40:49.708 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.709 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.709 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 7, 2020-03-11T22:40:49.709 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "room" 2020-03-11T22:40:49.709 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.709 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.709 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 8, 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "datacenter" 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 9, 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "zone" 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.710 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 10, 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "region" 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "root" 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.711 INFO:tasks.workunit.client.0.smithi130.stdout: "buckets": [ 2020-03-11T22:40:49.712 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.712 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -1, 2020-03-11T22:40:49.712 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default", 2020-03-11T22:40:49.712 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:40:49.712 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:40:49.712 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:40:49.712 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.713 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.714 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.714 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.714 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -2, 2020-03-11T22:40:49.714 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default~hdd", 2020-03-11T22:40:49.714 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:40:49.714 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:40:49.714 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:40:49.715 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130", 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:40:49.716 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:40:49.717 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:40:49.717 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:40:49.717 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:40:49.717 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.717 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:40:49.717 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:40:49.717 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.718 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:40:49.719 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130~hdd", 2020-03-11T22:40:49.720 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:40:49.720 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:40:49.720 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:40:49.720 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:40:49.720 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:40:49.720 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:40:49.720 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.721 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:40:49.721 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:40:49.721 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:40:49.721 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.721 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.721 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:40:49.722 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:40:49.722 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:40:49.722 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.722 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.722 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:40:49.722 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:40:49.722 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:40:49.723 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.724 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.724 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.724 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.725 INFO:tasks.workunit.client.0.smithi130.stdout: "rules": [ 2020-03-11T22:40:49.725 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.725 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 0, 2020-03-11T22:40:49.725 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "replicated_rule", 2020-03-11T22:40:49.725 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 0, 2020-03-11T22:40:49.725 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:40:49.725 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 1, 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 10, 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.726 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.727 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_firstn", 2020-03-11T22:40:49.727 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:40:49.727 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:40:49.727 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.727 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.727 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:40:49.727 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.728 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:40:49.728 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.728 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.728 INFO:tasks.workunit.client.0.smithi130.stdout: "tunables": { 2020-03-11T22:40:49.728 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_tries": 0, 2020-03-11T22:40:49.728 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:40:49.728 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_total_tries": 50, 2020-03-11T22:40:49.729 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:40:49.729 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:40:49.729 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_stable": 1, 2020-03-11T22:40:49.729 INFO:tasks.workunit.client.0.smithi130.stdout: "straw_calc_version": 1, 2020-03-11T22:40:49.729 INFO:tasks.workunit.client.0.smithi130.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:40:49.729 INFO:tasks.workunit.client.0.smithi130.stdout: "profile": "jewel", 2020-03-11T22:40:49.729 INFO:tasks.workunit.client.0.smithi130.stdout: "optimal_tunables": 1, 2020-03-11T22:40:49.730 INFO:tasks.workunit.client.0.smithi130.stdout: "legacy_tunables": 0, 2020-03-11T22:40:49.730 INFO:tasks.workunit.client.0.smithi130.stdout: "minimum_required_version": "jewel", 2020-03-11T22:40:49.730 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables": 1, 2020-03-11T22:40:49.730 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables2": 1, 2020-03-11T22:40:49.730 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v2_rules": 0, 2020-03-11T22:40:49.730 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables3": 1, 2020-03-11T22:40:49.730 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v3_rules": 0, 2020-03-11T22:40:49.731 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v4_buckets": 1, 2020-03-11T22:40:49.731 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables5": 1, 2020-03-11T22:40:49.731 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v5_rules": 0 2020-03-11T22:40:49.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.731 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_args": {} 2020-03-11T22:40:49.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.731 INFO:tasks.workunit.client.0.smithi130.stdout: "fsmap": { 2020-03-11T22:40:49.732 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:40:49.732 INFO:tasks.workunit.client.0.smithi130.stdout: "default_fscid": -1, 2020-03-11T22:40:49.732 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": { 2020-03-11T22:40:49.732 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": {}, 2020-03-11T22:40:49.732 INFO:tasks.workunit.client.0.smithi130.stdout: "ro_compat": {}, 2020-03-11T22:40:49.732 INFO:tasks.workunit.client.0.smithi130.stdout: "incompat": { 2020-03-11T22:40:49.732 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_1": "base v0.20", 2020-03-11T22:40:49.733 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:40:49.733 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:40:49.733 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:40:49.733 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:40:49.733 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:40:49.733 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_8": "no anchor table", 2020-03-11T22:40:49.733 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_9": "file layout v2", 2020-03-11T22:40:49.734 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:40:49.734 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.734 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.734 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_flags": { 2020-03-11T22:40:49.734 INFO:tasks.workunit.client.0.smithi130.stdout: "enable_multiple": false, 2020-03-11T22:40:49.734 INFO:tasks.workunit.client.0.smithi130.stdout: "ever_enabled_multiple": false 2020-03-11T22:40:49.734 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.735 INFO:tasks.workunit.client.0.smithi130.stdout: "standbys": [], 2020-03-11T22:40:49.735 INFO:tasks.workunit.client.0.smithi130.stdout: "filesystems": [] 2020-03-11T22:40:49.735 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.735 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:40:49.735 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:40:49.735 INFO:tasks.workunit.client.0.smithi130.stdout: "auth": { 2020-03-11T22:40:49.735 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:40:49.736 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 5, 2020-03-11T22:40:49.736 INFO:tasks.workunit.client.0.smithi130.stdout: "num_secrets": 3 2020-03-11T22:40:49.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.736 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 32, 2020-03-11T22:40:49.736 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_active": 32, 2020-03-11T22:40:49.736 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_unknown": 0, 2020-03-11T22:40:49.736 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osd": 3, 2020-03-11T22:40:49.737 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_sum": { 2020-03-11T22:40:49.737 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:40:49.737 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48300, 2020-03-11T22:40:49.737 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:40:49.737 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:40:49.737 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:40:49.737 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:49.738 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:40:49.738 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:49.738 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:49.738 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:49.738 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:49.738 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:40:49.738 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:40:49.739 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:40:49.739 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 100, 2020-03-11T22:40:49.739 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 100, 2020-03-11T22:40:49.739 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:49.739 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:49.739 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:49.739 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:40:49.740 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:40:49.741 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:40:49.741 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:40:49.741 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:49.741 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:49.741 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:49.741 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:49.741 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.742 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:40:49.743 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:40:49.743 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:40:49.743 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:40:49.743 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 0, 2020-03-11T22:40:49.743 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 0, 2020-03-11T22:40:49.743 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:40:49.743 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:49.744 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:40:49.744 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:40:49.744 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:40:49.744 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.744 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 512, 2020-03-11T22:40:49.744 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 512, 2020-03-11T22:40:49.745 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 64, 2020-03-11T22:40:49.745 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 64, 2020-03-11T22:40:49.745 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 0 2020-03-11T22:40:49.745 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.745 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum": { 2020-03-11T22:40:49.745 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:40:49.745 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 65, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3180352, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 34304, 2020-03-11T22:40:49.746 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 8, 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145719, 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311392448, 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318865866752, 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 35127296, 2020-03-11T22:40:49.747 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 432213, 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 9212, 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221216260 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:40:49.748 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:49.749 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:49.749 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:49.749 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:40:49.749 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:40:49.749 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:40:49.749 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.749 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:40:49.750 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.751 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum_by_class": { 2020-03-11T22:40:49.751 INFO:tasks.workunit.client.0.smithi130.stdout: "hdd": { 2020-03-11T22:40:49.751 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:40:49.751 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:40:49.751 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 65, 2020-03-11T22:40:49.751 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:40:49.751 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:49.752 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:49.752 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:40:49.752 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3180352, 2020-03-11T22:40:49.752 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 34304, 2020-03-11T22:40:49.752 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 8, 2020-03-11T22:40:49.752 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145719, 2020-03-11T22:40:49.752 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311392448, 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318865866752, 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 35127296, 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 432213, 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:40:49.753 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:49.754 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:40:49.754 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 9212, 2020-03-11T22:40:49.754 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221216260 2020-03-11T22:40:49.754 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.754 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:40:49.754 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:49.754 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:49.755 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:49.756 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:49.756 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:40:49.756 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.756 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:40:49.756 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:40:49.756 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.756 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_stats": [ 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: "poolid": 1, 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 32, 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48300, 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:40:49.757 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:49.758 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:40:49.759 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:40:49.759 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:40:49.759 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 100, 2020-03-11T22:40:49.759 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 100, 2020-03-11T22:40:49.759 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:49.759 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:49.759 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:49.760 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:49.760 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:49.760 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:49.760 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:40:49.760 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:49.760 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:49.760 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:49.762 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:49.762 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:49.762 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:49.762 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:49.763 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:49.763 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:40:49.763 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:40:49.763 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.763 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:40:49.763 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:40:49.763 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:40:49.764 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:40:49.764 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 19529728, 2020-03-11T22:40:49.764 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 143228, 2020-03-11T22:40:49.764 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:40:49.764 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:49.764 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:40:49.764 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 512, 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 512, 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 64, 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 64, 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 3 2020-03-11T22:40:49.765 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_stats": [ 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 154618822666 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.766 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:40:49.767 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 42949672975 2020-03-11T22:40:49.767 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.767 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.767 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:40:49.767 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 158913789961 2020-03-11T22:40:49.767 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.767 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.768 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_state": [ 2020-03-11T22:40:49.768 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.768 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2020-03-11T22:40:49.768 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 32 2020-03-11T22:40:49.768 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.768 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.768 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_osd": [ 2020-03-11T22:40:49.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.769 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:40:49.769 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:40:49.769 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 32, 2020-03-11T22:40:49.769 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:49.769 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.770 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:40:49.770 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:40:49.770 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 0, 2020-03-11T22:40:49.770 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:49.770 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.770 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.770 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 32, 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 32, 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [ 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:40:49.771 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:40:49.772 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": {} 2020-03-11T22:40:49.772 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.772 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:40:49.772 INFO:tasks.workunit.client.0.smithi130.stdout: "servicemap": { 2020-03-11T22:40:49.772 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:40:49.772 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:39:12.121431+0000", 2020-03-11T22:40:49.772 INFO:tasks.workunit.client.0.smithi130.stdout: "services": {} 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout: "paxos": { 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 316, 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pn": 0, 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout: "accepted_pn": 0 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:40:49.773 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:40:49.774 INFO:tasks.workunit.client.0.smithi130.stderr:report 687258196 2020-03-11T22:40:49.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:40:49.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:40:49.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:40:49.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:40:49.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:49.775 INFO:tasks.workunit.client.0.smithi130.stdout:writing non-divergent object existing_4 2020-03-11T22:40:49.912 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:49.913 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450273+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.913 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449796+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.913 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449577+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.913 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449598+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.913 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449442+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.913 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447316+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.914 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447315+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.914 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.914 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.914 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449034+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.914 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448575+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.914 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448248+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.914 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448747+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448753+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448664+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449024+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448659+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448335+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447984+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448289+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.915 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:30.756253+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.916 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449444+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.916 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449867+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.916 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447310+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.916 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447318+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.916 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447311+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.916 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450417+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.916 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447906+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.917 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447911+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.917 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448358+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.917 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447961+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.917 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449168+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:40:49.917 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:40:49.917 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:49.918 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:40:49.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:40:49.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:49.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:40:49.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:49.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:49.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:49.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:49.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:49.981 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:49.981 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:49.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:49.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:49.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:50.037 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:50.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:50.387 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:40:50.387 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:40:50.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:50.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:50.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:50.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822668 2020-03-11T22:40:50.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822668' 2020-03-11T22:40:50.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:50.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:50.609 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:50.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:50.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822668 1-' 2020-03-11T22:40:50.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:50.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:50.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789964 2020-03-11T22:40:50.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822668 1- 2-158913789964' 2020-03-11T22:40:50.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:50.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822668 2020-03-11T22:40:50.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:50.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:50.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822668 2020-03-11T22:40:50.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:50.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822668 2020-03-11T22:40:50.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822668' 2020-03-11T22:40:50.750 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 154618822668 2020-03-11T22:40:50.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822666 -lt 154618822668 2020-03-11T22:40:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:52.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:52.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:52.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822669 -lt 154618822668 2020-03-11T22:40:52.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:52.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:52.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:52.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:52.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:52.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:52.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:52.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:52.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:52.448 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 2020-03-11T22:40:52.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:40:52.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:40:52.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:52.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789964 2020-03-11T22:40:52.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:52.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:52.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789964 2020-03-11T22:40:52.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:52.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789964 2020-03-11T22:40:52.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789964' 2020-03-11T22:40:52.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:52.801 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 158913789964 2020-03-11T22:40:53.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789964 -lt 158913789964 2020-03-11T22:40:53.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:53.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:53.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:53.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:40:53.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:53.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:53.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:53.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:53.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:53.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:53.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:53.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:53.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:53.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:53.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:54.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:54.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:40:54.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:40:54.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:40:54.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:54.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:40:54.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:54.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:54.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:54.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:54.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:54.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:54.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:54.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:54.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:54.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:54.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:54.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:55.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:55.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:55.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:55.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:55.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:55.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:55.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:55.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:55.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:55.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:55.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:55.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:40:55.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:55.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:40:55.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:55.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:55.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:55.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:55.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:55.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:55.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:55.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:56.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:56.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:56.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:56.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:56.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:56.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:56.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:56.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:56.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:56.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:56.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:56.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:56.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:57.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:57.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:57.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:40:57.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:57.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:40:57.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:57.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:57.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:57.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:57.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:57.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:57.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:57.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:57.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:57.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:57.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:58.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:58.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:58.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:58.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:58.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:58.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:40:58.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:58.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:40:59.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:59.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:59.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:59.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:59.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:59.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:59.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:00.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:00.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:00.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:00.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:00.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:00.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:00.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:00.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:00.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:00.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:41:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:41:02.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:02.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:02.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:02.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:02.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:02.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:02.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:02.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:02.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:02.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:02.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:02.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:03.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:03.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:03.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:03.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:03.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:03.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:03.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:03.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:03.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:03.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:03.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:03.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:41:03.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:03.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:41:06.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:06.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:06.862 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:06.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:06.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:06.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:06.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:06.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:07.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:07.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:07.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:07.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:07.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:07.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:07.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:07.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:07.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:07.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:07.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:07.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:07.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:08.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:08.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:08.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:41:08.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:08.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:41:14.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:14.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:14.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:14.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:14.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:14.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:14.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:14.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:14.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:14.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:14.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:14.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:15.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:15.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:15.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:15.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:15.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:15.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:15.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:15.300 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:15.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:15.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:15.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:15.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:41:15.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:15.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:41:23.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:23.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:23.044 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:23.044 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:23.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:23.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:23.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:23.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:23.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:23.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:23.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:23.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:23.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:23.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:23.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:23.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:23.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:23.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:23.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:23.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:23.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:24.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:24.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:24.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:41:24.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:41:24.708 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:41:24.708 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_fingerprint": "0c682c5e-762b-4242-a76c-209ee2638d46", 2020-03-11T22:41:24.709 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:41:24.709 INFO:tasks.workunit.client.0.smithi130.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:41:24.709 INFO:tasks.workunit.client.0.smithi130.stdout: "timestamp": "2020-03-11T22:41:24.705324+0000", 2020-03-11T22:41:24.709 INFO:tasks.workunit.client.0.smithi130.stdout: "tag": "", 2020-03-11T22:41:24.709 INFO:tasks.workunit.client.0.smithi130.stdout: "health": { 2020-03-11T22:41:24.709 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "HEALTH_WARN", 2020-03-11T22:41:24.709 INFO:tasks.workunit.client.0.smithi130.stdout: "checks": { 2020-03-11T22:41:24.710 INFO:tasks.workunit.client.0.smithi130.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:41:24.710 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:24.710 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:41:24.710 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:41:24.710 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 17 2020-03-11T22:41:24.710 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.710 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [], 2020-03-11T22:41:24.711 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:41:24.711 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.711 INFO:tasks.workunit.client.0.smithi130.stdout: "OSD_DOWN": { 2020-03-11T22:41:24.711 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:24.711 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:41:24.711 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 osds down", 2020-03-11T22:41:24.711 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "osd.1 (root=default,host=smithi130) is down" 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:41:24.712 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.713 INFO:tasks.workunit.client.0.smithi130.stdout: "PG_DEGRADED": { 2020-03-11T22:41:24.713 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:24.713 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:41:24.713 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:41:24.713 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 64 2020-03-11T22:41:24.713 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.713 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:41:24.714 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.714 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.0 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.714 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.714 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.714 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.714 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.714 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.2 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.3 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.4 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.715 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.716 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.716 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.5 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.716 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.716 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.716 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.6 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.716 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.716 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.717 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.7 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.717 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.718 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.718 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.8 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.718 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.718 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.718 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.9 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.718 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.718 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.719 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.a is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.719 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.719 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.724 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.b is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.724 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.724 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.725 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.c is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.725 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.725 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.725 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.d is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.725 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.725 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.725 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.e is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.726 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.726 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.726 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.f is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.726 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.726 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.726 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.10 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.11 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.12 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.727 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.728 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.13 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.728 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.728 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.728 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.14 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.728 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.728 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.728 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.15 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.729 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.729 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.729 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.16 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.729 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.729 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.729 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.17 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.729 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.18 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.19 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.730 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1a is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.731 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.731 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1b is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.731 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.731 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1c is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.732 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.732 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1d is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.732 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1e is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.1f is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.735 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.735 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:41:24.735 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.735 INFO:tasks.workunit.client.0.smithi130.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:41:24.735 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:24.735 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.736 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.737 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:41:24.737 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.737 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.737 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:41:24.737 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.737 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.738 INFO:tasks.workunit.client.0.smithi130.stdout: "mutes": [] 2020-03-11T22:41:24.738 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.738 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_first_committed": 1, 2020-03-11T22:41:24.738 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_last_committed": 1, 2020-03-11T22:41:24.738 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap": { 2020-03-11T22:41:24.738 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:41:24.738 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:41:24.739 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:39:11.426694Z", 2020-03-11T22:41:24.739 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:39:11.426694Z", 2020-03-11T22:41:24.739 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release": 15, 2020-03-11T22:41:24.739 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:41:24.739 INFO:tasks.workunit.client.0.smithi130.stdout: "features": { 2020-03-11T22:41:24.740 INFO:tasks.workunit.client.0.smithi130.stdout: "persistent": [ 2020-03-11T22:41:24.740 INFO:tasks.workunit.client.0.smithi130.stdout: "kraken", 2020-03-11T22:41:24.740 INFO:tasks.workunit.client.0.smithi130.stdout: "luminous", 2020-03-11T22:41:24.740 INFO:tasks.workunit.client.0.smithi130.stdout: "mimic", 2020-03-11T22:41:24.740 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap-prune", 2020-03-11T22:41:24.740 INFO:tasks.workunit.client.0.smithi130.stdout: "nautilus", 2020-03-11T22:41:24.740 INFO:tasks.workunit.client.0.smithi130.stdout: "octopus" 2020-03-11T22:41:24.741 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.741 INFO:tasks.workunit.client.0.smithi130.stdout: "optional": [] 2020-03-11T22:41:24.741 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.741 INFO:tasks.workunit.client.0.smithi130.stdout: "mons": [ 2020-03-11T22:41:24.741 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.741 INFO:tasks.workunit.client.0.smithi130.stdout: "rank": 0, 2020-03-11T22:41:24.742 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "a", 2020-03-11T22:41:24.742 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:41:24.742 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.742 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.742 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.742 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:41:24.742 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 0 2020-03-11T22:41:24.743 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.743 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.743 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.743 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:41:24.743 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:41:24.743 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 0, 2020-03-11T22:41:24.743 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 0 2020-03-11T22:41:24.744 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.744 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.744 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.744 INFO:tasks.workunit.client.0.smithi130.stdout: "quorum": [ 2020-03-11T22:41:24.744 INFO:tasks.workunit.client.0.smithi130.stdout: 0 2020-03-11T22:41:24.744 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap": { 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 43, 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:39:12.120944+0000", 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:40:18.133240+0000", 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "last_up_change": "2020-03-11T22:40:13.149407+0000", 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "last_in_change": "2020-03-11T22:40:14.362860+0000", 2020-03-11T22:41:24.745 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:41:24.746 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_num": 5800384, 2020-03-11T22:41:24.746 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_set": [ 2020-03-11T22:41:24.746 INFO:tasks.workunit.client.0.smithi130.stdout: "nodown", 2020-03-11T22:41:24.746 INFO:tasks.workunit.client.0.smithi130.stdout: "noin", 2020-03-11T22:41:24.746 INFO:tasks.workunit.client.0.smithi130.stdout: "noout", 2020-03-11T22:41:24.746 INFO:tasks.workunit.client.0.smithi130.stdout: "pglog_hardlimit", 2020-03-11T22:41:24.746 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snapdirs", 2020-03-11T22:41:24.747 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_deletes", 2020-03-11T22:41:24.747 INFO:tasks.workunit.client.0.smithi130.stdout: "sortbitwise" 2020-03-11T22:41:24.747 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.747 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_version": 7, 2020-03-11T22:41:24.747 INFO:tasks.workunit.client.0.smithi130.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:41:24.747 INFO:tasks.workunit.client.0.smithi130.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:41:24.747 INFO:tasks.workunit.client.0.smithi130.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:41:24.748 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_snapshot": "", 2020-03-11T22:41:24.748 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_max": 1, 2020-03-11T22:41:24.748 INFO:tasks.workunit.client.0.smithi130.stdout: "max_osd": 3, 2020-03-11T22:41:24.748 INFO:tasks.workunit.client.0.smithi130.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:41:24.748 INFO:tasks.workunit.client.0.smithi130.stdout: "min_compat_client": "jewel", 2020-03-11T22:41:24.748 INFO:tasks.workunit.client.0.smithi130.stdout: "require_osd_release": "octopus", 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: "pools": [ 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_name": "test", 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: "create_time": "2020-03-11T22:39:33.194196+0000", 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": 1, 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_names": "hashpspool", 2020-03-11T22:41:24.749 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:41:24.750 INFO:tasks.workunit.client.0.smithi130.stdout: "size": 3, 2020-03-11T22:41:24.750 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 2, 2020-03-11T22:41:24.750 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_rule": 0, 2020-03-11T22:41:24.750 INFO:tasks.workunit.client.0.smithi130.stdout: "object_hash": 2, 2020-03-11T22:41:24.750 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:41:24.750 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num": 32, 2020-03-11T22:41:24.750 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num": 1, 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num_target": 32, 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_target": 32, 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_pending": 32, 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pg_merge_meta": { 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "source_pgid": "0.0", 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "ready_epoch": 0, 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 0, 2020-03-11T22:41:24.751 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 0, 2020-03-11T22:41:24.752 INFO:tasks.workunit.client.0.smithi130.stdout: "source_version": "0'0", 2020-03-11T22:41:24.752 INFO:tasks.workunit.client.0.smithi130.stdout: "target_version": "0'0" 2020-03-11T22:41:24.752 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.752 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "42", 2020-03-11T22:41:24.752 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend": "0", 2020-03-11T22:41:24.752 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:41:24.752 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_preluminous": "42", 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "auid": 0, 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_seq": 0, 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_epoch": 0, 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_snaps": [], 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps": "[]", 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_bytes": 0, 2020-03-11T22:41:24.753 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_objects": 0, 2020-03-11T22:41:24.754 INFO:tasks.workunit.client.0.smithi130.stdout: "tiers": [], 2020-03-11T22:41:24.754 INFO:tasks.workunit.client.0.smithi130.stdout: "tier_of": -1, 2020-03-11T22:41:24.754 INFO:tasks.workunit.client.0.smithi130.stdout: "read_tier": -1, 2020-03-11T22:41:24.754 INFO:tasks.workunit.client.0.smithi130.stdout: "write_tier": -1, 2020-03-11T22:41:24.754 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_mode": "none", 2020-03-11T22:41:24.754 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_bytes": 0, 2020-03-11T22:41:24.754 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_objects": 0, 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_flush_age": 0, 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_evict_age": 0, 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profile": "", 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_params": { 2020-03-11T22:41:24.755 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "none" 2020-03-11T22:41:24.756 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.756 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_period": 0, 2020-03-11T22:41:24.756 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_count": 0, 2020-03-11T22:41:24.756 INFO:tasks.workunit.client.0.smithi130.stdout: "use_gmt_hitset": true, 2020-03-11T22:41:24.756 INFO:tasks.workunit.client.0.smithi130.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:41:24.756 INFO:tasks.workunit.client.0.smithi130.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:41:24.756 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:41:24.757 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:41:24.757 INFO:tasks.workunit.client.0.smithi130.stdout: "grade_table": [], 2020-03-11T22:41:24.757 INFO:tasks.workunit.client.0.smithi130.stdout: "stripe_width": 0, 2020-03-11T22:41:24.757 INFO:tasks.workunit.client.0.smithi130.stdout: "expected_num_objects": 0, 2020-03-11T22:41:24.757 INFO:tasks.workunit.client.0.smithi130.stdout: "fast_read": false, 2020-03-11T22:41:24.757 INFO:tasks.workunit.client.0.smithi130.stdout: "options": {}, 2020-03-11T22:41:24.757 INFO:tasks.workunit.client.0.smithi130.stdout: "application_metadata": {} 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: "osds": [ 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "dad61a67-7946-4e56-97b6-52baacd3a88d", 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:41:24.758 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:41:24.759 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:41:24.759 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:41:24.759 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 5, 2020-03-11T22:41:24.759 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 30, 2020-03-11T22:41:24.759 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 36, 2020-03-11T22:41:24.759 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 42, 2020-03-11T22:41:24.759 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 31, 2020-03-11T22:41:24.760 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:41:24.760 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:41:24.760 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.760 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.760 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.760 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:41:24.760 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.761 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.761 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.761 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.761 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:41:24.761 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.761 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.761 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.762 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.762 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:41:24.762 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.762 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.762 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.762 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:41:24.762 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.763 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.764 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:24.764 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.764 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.764 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.764 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:41:24.764 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.764 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.765 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.765 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.765 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:41:24.765 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.765 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.765 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.765 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39310 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6803/39310", 2020-03-11T22:41:24.767 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6805/39310", 2020-03-11T22:41:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39310", 2020-03-11T22:41:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39310", 2020-03-11T22:41:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:41:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:41:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:41:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "53637eac-f654-4841-82d1-b00b8684e01b", 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 0, 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:41:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 0, 2020-03-11T22:41:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 0, 2020-03-11T22:41:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 10, 2020-03-11T22:41:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 22, 2020-03-11T22:41:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 32, 2020-03-11T22:41:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:41:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:41:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.771 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:24.772 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.772 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.772 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.772 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.772 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:41:24.772 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.772 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:24.773 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.774 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.774 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.774 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.774 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:24.774 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.774 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.774 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.775 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.776 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.776 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.776 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:24.776 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.776 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.776 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.776 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:24.777 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.777 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.777 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.777 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.777 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:24.777 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 34884 2020-03-11T22:41:24.778 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.778 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.778 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.778 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/34884", 2020-03-11T22:41:24.778 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/34884", 2020-03-11T22:41:24.778 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34884", 2020-03-11T22:41:24.779 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34884", 2020-03-11T22:41:24.779 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:41:24.779 INFO:tasks.workunit.client.0.smithi130.stdout: "exists" 2020-03-11T22:41:24.779 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.779 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.779 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.780 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:41:24.780 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "41240656-1515-44bf-9b8c-f85a0a4cd9c5", 2020-03-11T22:41:24.780 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:41:24.780 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:41:24.780 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:41:24.780 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:41:24.780 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 15, 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 30, 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 37, 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 0, 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 33, 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.781 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.782 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.782 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:24.782 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.782 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.782 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.782 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.782 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:24.783 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.783 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.783 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.783 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.783 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:41:24.783 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.783 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:24.784 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.785 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.785 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.785 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.785 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:24.785 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.785 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.785 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.786 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.787 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.787 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.787 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:24.787 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:41:24.787 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.787 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:41:24.787 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:24.788 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.788 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.788 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.788 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:41:24.788 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:24.788 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 39504 2020-03-11T22:41:24.788 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/39504", 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/39504", 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39504", 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39504", 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:41:24.789 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:41:24.790 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:41:24.790 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.790 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.790 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.790 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_xinfo": [ 2020-03-11T22:41:24.790 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.790 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:40:06.831157+0000", 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:18.723697+0000", 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:41:24.791 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.792 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.792 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:41:24.792 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:40:07.386483+0000", 2020-03-11T22:41:24.792 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:41:24.792 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:41:24.792 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:41:24.792 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:41:24.793 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:24.974450+0000", 2020-03-11T22:41:24.793 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:41:24.793 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.793 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.793 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:41:24.793 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:40:07.920971+0000", 2020-03-11T22:41:24.793 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:31.102932+0000", 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.794 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap": [], 2020-03-11T22:41:24.795 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap_items": [], 2020-03-11T22:41:24.795 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_temp": [], 2020-03-11T22:41:24.795 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_temp": [], 2020-03-11T22:41:24.795 INFO:tasks.workunit.client.0.smithi130.stdout: "blacklist": {}, 2020-03-11T22:41:24.795 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profiles": { 2020-03-11T22:41:24.795 INFO:tasks.workunit.client.0.smithi130.stdout: "default": { 2020-03-11T22:41:24.795 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "2", 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van" 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps_queue": [], 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: "new_removed_snaps": [], 2020-03-11T22:41:24.796 INFO:tasks.workunit.client.0.smithi130.stdout: "new_purged_snaps": [], 2020-03-11T22:41:24.797 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_node_flags": {}, 2020-03-11T22:41:24.797 INFO:tasks.workunit.client.0.smithi130.stdout: "device_class_flags": {} 2020-03-11T22:41:24.797 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.797 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_metadata": [ 2020-03-11T22:41:24.797 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.797 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:41:24.797 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:41:24.798 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6804/39310,v1:127.0.0.1:6805/39310]", 2020-03-11T22:41:24.799 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:41:24.799 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:41:24.799 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:24.799 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:24.799 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:24.799 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:24.799 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:24.800 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:24.800 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:24.800 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:41:24.800 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:24.800 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:24.800 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:24.800 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:24.801 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:41:24.801 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:24.801 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:24.801 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:24.801 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:41:24.801 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:41:24.801 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:24.802 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:41:24.802 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:41:24.802 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:24.802 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:41:24.802 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6802/39310,v1:127.0.0.1:6803/39310]", 2020-03-11T22:41:24.802 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:41:24.802 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39310,v1:127.0.0.1:6809/39310]", 2020-03-11T22:41:24.803 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39310,v1:127.0.0.1:6807/39310]", 2020-03-11T22:41:24.803 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:41:24.803 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:41:24.803 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:24.803 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:24.803 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:24.803 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:41:24.804 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:24.804 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:24.804 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:41:24.804 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:41:24.804 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:24.804 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:41:24.804 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/34884,v1:127.0.0.1:6813/34884]", 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:24.805 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:24.806 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:24.806 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:24.806 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:24.806 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:24.806 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:24.806 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:41:24.806 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:24.807 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:24.807 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:24.807 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:24.807 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:41:24.807 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:24.807 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:24.807 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:41:24.808 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/34884,v1:127.0.0.1:6811/34884]", 2020-03-11T22:41:24.809 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:41:24.809 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34884,v1:127.0.0.1:6817/34884]", 2020-03-11T22:41:24.809 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34884,v1:127.0.0.1:6815/34884]", 2020-03-11T22:41:24.809 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:41:24.809 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:41:24.809 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:24.809 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:24.810 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/39504,v1:127.0.0.1:6813/39504]", 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:41:24.811 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:24.812 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:24.813 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:24.814 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:24.814 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:41:24.814 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:41:24.814 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:24.814 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:41:24.814 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:41:24.814 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:24.815 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:41:24.815 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/39504,v1:127.0.0.1:6811/39504]", 2020-03-11T22:41:24.815 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:41:24.815 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39504,v1:127.0.0.1:6817/39504]", 2020-03-11T22:41:24.815 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39504,v1:127.0.0.1:6815/39504]", 2020-03-11T22:41:24.815 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:41:24.815 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:41:24.816 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:41:24.817 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:24.817 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:41:24.817 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.817 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.817 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_first_committed": 1, 2020-03-11T22:41:24.817 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_last_committed": 43, 2020-03-11T22:41:24.817 INFO:tasks.workunit.client.0.smithi130.stdout: "crushmap": { 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": [ 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.0", 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.818 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:41:24.819 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.1", 2020-03-11T22:41:24.819 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:41:24.819 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.819 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.819 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:41:24.819 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.2", 2020-03-11T22:41:24.819 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: "types": [ 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 0, 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd" 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.820 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.821 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:41:24.821 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "host" 2020-03-11T22:41:24.821 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.821 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.821 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 2, 2020-03-11T22:41:24.821 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "chassis" 2020-03-11T22:41:24.821 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 3, 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "rack" 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 4, 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "row" 2020-03-11T22:41:24.822 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.823 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.823 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 5, 2020-03-11T22:41:24.823 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pdu" 2020-03-11T22:41:24.823 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.823 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.823 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 6, 2020-03-11T22:41:24.823 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pod" 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 7, 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "room" 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 8, 2020-03-11T22:41:24.824 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "datacenter" 2020-03-11T22:41:24.825 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.825 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.825 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 9, 2020-03-11T22:41:24.825 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "zone" 2020-03-11T22:41:24.825 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.825 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.825 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 10, 2020-03-11T22:41:24.826 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "region" 2020-03-11T22:41:24.826 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.826 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.826 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:41:24.826 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "root" 2020-03-11T22:41:24.826 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.826 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: "buckets": [ 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -1, 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default", 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:41:24.827 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:41:24.828 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:41:24.828 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:41:24.828 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.828 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:41:24.828 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:41:24.828 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:41:24.828 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -2, 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default~hdd", 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:41:24.829 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:41:24.830 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:41:24.830 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:41:24.830 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:41:24.830 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.830 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:41:24.830 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:41:24.830 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130", 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:41:24.831 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:41:24.832 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:41:24.832 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:41:24.832 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:41:24.832 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:41:24.832 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.832 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:41:24.832 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:41:24.833 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:41:24.833 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.833 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.833 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:41:24.833 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:41:24.833 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:41:24.833 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.834 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.835 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:41:24.835 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130~hdd", 2020-03-11T22:41:24.835 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:41:24.835 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:41:24.835 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:41:24.835 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:41:24.835 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.836 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:41:24.837 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:41:24.837 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:41:24.837 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.837 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.837 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:41:24.837 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:41:24.837 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:41:24.838 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.838 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.838 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.838 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.838 INFO:tasks.workunit.client.0.smithi130.stdout: "rules": [ 2020-03-11T22:41:24.838 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.838 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 0, 2020-03-11T22:41:24.839 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "replicated_rule", 2020-03-11T22:41:24.839 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 0, 2020-03-11T22:41:24.839 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:41:24.839 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 1, 2020-03-11T22:41:24.839 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 10, 2020-03-11T22:41:24.839 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:41:24.839 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.840 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:41:24.840 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:41:24.840 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:41:24.840 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.841 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.841 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_firstn", 2020-03-11T22:41:24.841 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:41:24.841 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:41:24.841 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.841 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.842 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:41:24.842 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.842 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:41:24.842 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.842 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.842 INFO:tasks.workunit.client.0.smithi130.stdout: "tunables": { 2020-03-11T22:41:24.842 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_tries": 0, 2020-03-11T22:41:24.843 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:41:24.843 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_total_tries": 50, 2020-03-11T22:41:24.843 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:41:24.843 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:41:24.843 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_stable": 1, 2020-03-11T22:41:24.843 INFO:tasks.workunit.client.0.smithi130.stdout: "straw_calc_version": 1, 2020-03-11T22:41:24.843 INFO:tasks.workunit.client.0.smithi130.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:41:24.844 INFO:tasks.workunit.client.0.smithi130.stdout: "profile": "jewel", 2020-03-11T22:41:24.844 INFO:tasks.workunit.client.0.smithi130.stdout: "optimal_tunables": 1, 2020-03-11T22:41:24.844 INFO:tasks.workunit.client.0.smithi130.stdout: "legacy_tunables": 0, 2020-03-11T22:41:24.844 INFO:tasks.workunit.client.0.smithi130.stdout: "minimum_required_version": "jewel", 2020-03-11T22:41:24.844 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables": 1, 2020-03-11T22:41:24.844 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables2": 1, 2020-03-11T22:41:24.844 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v2_rules": 0, 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables3": 1, 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v3_rules": 0, 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v4_buckets": 1, 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables5": 1, 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v5_rules": 0 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_args": {} 2020-03-11T22:41:24.845 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.846 INFO:tasks.workunit.client.0.smithi130.stdout: "fsmap": { 2020-03-11T22:41:24.846 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:41:24.846 INFO:tasks.workunit.client.0.smithi130.stdout: "default_fscid": -1, 2020-03-11T22:41:24.846 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": { 2020-03-11T22:41:24.846 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": {}, 2020-03-11T22:41:24.846 INFO:tasks.workunit.client.0.smithi130.stdout: "ro_compat": {}, 2020-03-11T22:41:24.846 INFO:tasks.workunit.client.0.smithi130.stdout: "incompat": { 2020-03-11T22:41:24.847 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_1": "base v0.20", 2020-03-11T22:41:24.847 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:41:24.847 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:41:24.847 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:41:24.847 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:41:24.847 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:41:24.847 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_8": "no anchor table", 2020-03-11T22:41:24.848 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_9": "file layout v2", 2020-03-11T22:41:24.848 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:41:24.848 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.848 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.848 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_flags": { 2020-03-11T22:41:24.848 INFO:tasks.workunit.client.0.smithi130.stdout: "enable_multiple": false, 2020-03-11T22:41:24.849 INFO:tasks.workunit.client.0.smithi130.stdout: "ever_enabled_multiple": false 2020-03-11T22:41:24.849 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.849 INFO:tasks.workunit.client.0.smithi130.stdout: "standbys": [], 2020-03-11T22:41:24.850 INFO:tasks.workunit.client.0.smithi130.stdout: "filesystems": [] 2020-03-11T22:41:24.850 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.850 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:41:24.850 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:41:24.850 INFO:tasks.workunit.client.0.smithi130.stdout: "auth": { 2020-03-11T22:41:24.850 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:41:24.850 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 5, 2020-03-11T22:41:24.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_secrets": 3 2020-03-11T22:41:24.851 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 32, 2020-03-11T22:41:24.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_active": 32, 2020-03-11T22:41:24.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_unknown": 0, 2020-03-11T22:41:24.851 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osd": 3, 2020-03-11T22:41:24.851 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_sum": { 2020-03-11T22:41:24.852 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:41:24.852 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 47947, 2020-03-11T22:41:24.852 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:41:24.852 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:41:24.852 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:41:24.852 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:24.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:41:24.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:24.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:24.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:24.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:24.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:41:24.853 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:24.854 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:24.855 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:24.855 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:41:24.855 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:24.855 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:24.855 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:41:24.855 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:41:24.855 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:24.856 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:24.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:24.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:24.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:24.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:41:24.857 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:41:24.857 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.857 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:41:24.858 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:41:24.858 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:41:24.858 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:41:24.858 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 0, 2020-03-11T22:41:24.858 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 0, 2020-03-11T22:41:24.858 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:41:24.859 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:24.859 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:41:24.859 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:41:24.859 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:41:24.859 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.859 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 513, 2020-03-11T22:41:24.859 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 513, 2020-03-11T22:41:24.860 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 64, 2020-03-11T22:41:24.860 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 64, 2020-03-11T22:41:24.860 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 0 2020-03-11T22:41:24.860 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.861 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum": { 2020-03-11T22:41:24.861 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:41:24.861 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:41:24.861 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 65, 2020-03-11T22:41:24.861 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:41:24.862 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:24.862 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:24.862 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:41:24.862 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3180352, 2020-03-11T22:41:24.862 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 34304, 2020-03-11T22:41:24.862 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 8, 2020-03-11T22:41:24.862 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145719, 2020-03-11T22:41:24.863 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311392448, 2020-03-11T22:41:24.863 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:41:24.863 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:41:24.863 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318865866752, 2020-03-11T22:41:24.863 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:24.863 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 35127296, 2020-03-11T22:41:24.863 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 431507, 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 9212, 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221216260 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:41:24.864 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:24.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:24.865 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:24.865 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:41:24.865 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:41:24.865 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:41:24.865 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.865 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:41:24.866 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.867 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum_by_class": { 2020-03-11T22:41:24.867 INFO:tasks.workunit.client.0.smithi130.stdout: "hdd": { 2020-03-11T22:41:24.867 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:41:24.867 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:41:24.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 65, 2020-03-11T22:41:24.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:41:24.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:24.868 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:24.868 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3180352, 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 34304, 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 8, 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145719, 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311392448, 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318865866752, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 35127296, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 431507, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:41:24.870 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 9212, 2020-03-11T22:41:24.871 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221216260 2020-03-11T22:41:24.871 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.871 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:41:24.871 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:24.871 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:24.871 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:24.871 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:24.872 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:41:24.873 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.873 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:41:24.873 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:41:24.873 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.873 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.873 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_stats": [ 2020-03-11T22:41:24.873 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.874 INFO:tasks.workunit.client.0.smithi130.stdout: "poolid": 1, 2020-03-11T22:41:24.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 32, 2020-03-11T22:41:24.874 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:41:24.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 47947, 2020-03-11T22:41:24.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 100, 2020-03-11T22:41:24.874 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2020-03-11T22:41:24.875 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 300, 2020-03-11T22:41:24.875 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:24.875 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:41:24.875 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:24.876 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:24.876 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:24.876 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:24.876 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:41:24.876 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:41:24.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:41:24.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:41:24.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:41:24.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:24.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:24.877 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:24.878 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:24.878 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:24.878 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:24.878 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:41:24.880 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:24.881 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:24.881 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:41:24.881 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:41:24.881 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:41:24.881 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:41:24.881 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:41:24.881 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:24.882 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:24.882 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:24.882 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:24.882 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:24.882 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:24.882 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:24.882 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:41:24.883 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 19529728, 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 142522, 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:41:24.884 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.885 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 513, 2020-03-11T22:41:24.885 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 513, 2020-03-11T22:41:24.885 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 64, 2020-03-11T22:41:24.885 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 64, 2020-03-11T22:41:24.885 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 3 2020-03-11T22:41:24.885 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.885 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_stats": [ 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 154618822675 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:41:24.886 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 42949672975 2020-03-11T22:41:24.887 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.887 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.887 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:41:24.887 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 158913789971 2020-03-11T22:41:24.887 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.887 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.887 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_state": [ 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 32 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_osd": [ 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.888 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:41:24.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:41:24.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 32, 2020-03-11T22:41:24.889 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:24.889 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.889 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.889 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 0, 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 32, 2020-03-11T22:41:24.890 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 32, 2020-03-11T22:41:24.891 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:24.891 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.891 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.891 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [ 2020-03-11T22:41:24.891 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:41:24.891 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:41:24.891 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": {} 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: "servicemap": { 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:39:12.121431+0000", 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: "services": {} 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:41:24.892 INFO:tasks.workunit.client.0.smithi130.stdout: "paxos": { 2020-03-11T22:41:24.893 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:41:24.893 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 396, 2020-03-11T22:41:24.893 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pn": 0, 2020-03-11T22:41:24.893 INFO:tasks.workunit.client.0.smithi130.stdout: "accepted_pn": 0 2020-03-11T22:41:24.893 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:41:24.893 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:41:24.894 INFO:tasks.workunit.client.0.smithi130.stderr:report 534656043 2020-03-11T22:41:24.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:41:24.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:41:24.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:24.895 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery 2020-03-11T22:41:25.041 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:25.041 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450273+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.041 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449796+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.041 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449577+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449598+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449442+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447316+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447315+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449034+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448575+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448248+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448747+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448753+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448664+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449024+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448659+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.043 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448335+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.044 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447984+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.044 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448289+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.044 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:30.756253+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.044 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449444+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.044 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449867+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.044 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447310+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.044 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447318+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.045 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447311+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.045 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450417+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.045 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447906+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.045 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447911+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.045 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448358+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.045 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:18.447961+0000 43'101 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.045 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449168+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.046 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:41:25.046 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:25.046 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:41:25.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:25.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:41:25.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:41:25.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:41:25.052 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:25.052 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:25.052 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:41:25.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:41:25.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:25.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:41:25.143 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:25.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:25.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:41:25.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:41:25.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:41:25.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:25.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:25.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:41:25.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:41:25.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:25.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:41:25.244 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:25.251 INFO:tasks.workunit.client.0.smithi130.stdout:revive divergent 1 2020-03-11T22:41:25.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:41:25.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:25.568 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:25.568 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450273+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.569 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449796+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.569 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449577+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.569 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449598+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.569 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449442+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.569 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447316+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.569 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447315+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449034+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448575+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448248+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448747+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448753+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.570 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448664+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.571 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449024+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.571 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448659+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.571 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448335+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.571 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447984+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.571 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448289+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.571 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:30.756253+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.571 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449444+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.572 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449867+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.572 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447310+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.572 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447318+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.572 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447311+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.572 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450417+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.572 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447906+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.572 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447911+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.573 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448358+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.573 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:18.447961+0000 43'101 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.573 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449168+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:25.573 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:41:25.573 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:25.573 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:41:25.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:41:26.206 INFO:tasks.workunit.client.0.smithi130.stderr:noup is set 2020-03-11T22:41:26.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:41:26.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:26.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:26.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:26.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:26.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:26.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:26.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:26.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:26.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:26.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:26.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:26.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:26.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:26.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:26.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:26.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:26.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:26.219 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:26.222 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:41:26.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:26.243 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:26.241+0000 7f01d8f09ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:26.243+0000 7f01d8f09ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:26.245+0000 7f01d8f09ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:26.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:41:26.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:41:26.763 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:26.760+0000 7f01d8f09ec0 -1 Falling back to public interface 2020-03-11T22:41:27.608 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:27.605+0000 7f01d8f09ec0 -1 osd.1 30 log_to_monitors {default=true} 2020-03-11T22:41:31.568 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery divergent 2020-03-11T22:41:31.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:41:31.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:31.900 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:31.900 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450273+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.900 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449796+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449577+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449598+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449442+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447316+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447315+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449173+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449034+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448575+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448248+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448747+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448753+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448664+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449024+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448659+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448335+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447984+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448289+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:30.756253+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449444+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449867+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447310+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447318+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447311+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.450417+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447906+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.447911+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.448358+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.905 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:18.447961+0000 43'101 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.905 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:18.449168+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:31.905 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:41:31.905 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:31.905 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:41:31.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:41:31.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:41:31.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:41:31.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:31.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:31.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:41:31.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:41:31.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:31.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:41:32.002 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:41:32.803 INFO:tasks.workunit.client.0.smithi130.stderr:noup is unset 2020-03-11T22:41:32.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:41:32.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:32.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:32.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:32.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:32.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:32.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:32.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:32.812 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:41:32.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:33.195 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 36 up_thru 42 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39310,v1:127.0.0.1:6803/39310] [v2:127.0.0.1:6804/39310,v1:127.0.0.1:6805/39310] exists,up dad61a67-7946-4e56-97b6-52baacd3a88d 2020-03-11T22:41:33.195 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:41:33.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:33.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:33.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:33.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:41:33.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:33.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:33.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:33.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:33.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:33.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:33.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:33.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:33.538 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42052,v1:127.0.0.1:6819/42052] [v2:127.0.0.1:6820/42052,v1:127.0.0.1:6821/42052] exists,up 53637eac-f654-4841-82d1-b00b8684e01b 2020-03-11T22:41:33.538 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:41:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:41:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:33.892 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,30) [v2:127.0.0.1:6810/39504,v1:127.0.0.1:6811/39504] [v2:127.0.0.1:6812/39504,v1:127.0.0.1:6813/39504] exists,up 41240656-1515-44bf-9b8c-f85a0a4cd9c5 2020-03-11T22:41:33.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:33.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:33.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:33.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:34.213 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:34.213 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.598940+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.213 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600350+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.213 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599052+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599010+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.598940+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.603196+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.603184+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599653+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599745+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599617+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.215 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601883+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.215 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602462+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.215 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600298+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.215 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600353+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.215 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600299+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.215 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601025+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.215 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601673+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.216 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.600957+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.216 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.601671+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.216 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.601003+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.216 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599679+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.216 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.600350+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.216 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599663+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.216 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602572+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.217 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602441+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.217 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602395+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.217 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.598994+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.217 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602463+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.217 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601946+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.217 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601065+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.217 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601617+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.218 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601022+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.218 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:41:34.218 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:34.218 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:41:34.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:41:34.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:34.223 INFO:tasks.workunit.client.0.smithi130.stdout:wait for peering 2020-03-11T22:41:34.545 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:34.546 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.598940+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.546 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600350+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.546 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599052+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.546 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599010+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.546 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.598940+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.546 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.603196+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.547 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.603184+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.547 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599653+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.547 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599745+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.547 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599617+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.547 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601883+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.547 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602462+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.547 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600298+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.548 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600353+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.548 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.600299+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.548 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601025+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.548 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601673+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.548 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.600957+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.548 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.601671+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.548 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.601003+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.549 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599679+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.549 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:41:32.600350+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.549 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.599663+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.549 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602572+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.549 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602441+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.549 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602395+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.550 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.598994+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.550 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.602463+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.550 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601946+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.550 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601065+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.550 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601617+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.550 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:41:32.601022+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:34.551 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:41:34.551 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:34.551 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:41:34.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:41:46.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:41:46.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:46.481 INFO:tasks.workunit.client.0.smithi130.stdout:killing divergent 1 2020-03-11T22:41:46.813 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:46.813 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.110563+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.813 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.107568+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.092166+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.106948+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:41:33.109439+0000 30'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108502+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.089047+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.091968+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.814 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.106137+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.109243+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.109570+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.089783+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.091673+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.090936+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:41:33.108864+0000 30'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.109769+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.815 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.106511+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.816 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.091392+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.816 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.089214+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.816 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108708+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.816 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.107816+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.816 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:41:33.109979+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.816 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.110310+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.816 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108260+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.817 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.088957+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.817 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:41:33.092512+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.817 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108181+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.817 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.109303+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.817 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:41:33.108502+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.817 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:41:33.090485+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.817 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:41:33.091079+0000 43'101 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.818 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.107236+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:46.818 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:41:46.818 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:46.818 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:41:46.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:41:46.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:46.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:46.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:46.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:46.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:46.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:46.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17424.out 2020-03-11T22:41:46.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17424.out 2020-03-11T22:41:46.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:46.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:46.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:46.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:46.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:46.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17424.out 2020-03-11T22:41:47.529 INFO:tasks.workunit.client.0.smithi130.stderr:Exporting 1.0 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/49 n=3 ec=22/22 lis/c=47/47 les/c/f=49/49/0 sis=47 pruub=13.302053139s) 2020-03-11T22:41:47.543 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:47.562 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:47.575 INFO:tasks.workunit.client.0.smithi130.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:47.586 INFO:tasks.workunit.client.0.smithi130.stderr:Export successful 2020-03-11T22:41:47.587 INFO:tasks.workunit.client.0.smithi130.stdout: marking collection for removal 2020-03-11T22:41:47.587 INFO:tasks.workunit.client.0.smithi130.stdout:setting '_remove' omap key 2020-03-11T22:41:47.587 INFO:tasks.workunit.client.0.smithi130.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:41:47.608 INFO:tasks.workunit.client.0.smithi130.stderr:Remove successful 2020-03-11T22:41:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17424.out 2020-03-11T22:41:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:48.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:48.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:48.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17424.out 2020-03-11T22:41:48.557 INFO:tasks.workunit.client.0.smithi130.stdout:get_pg_num_history pg_num_history pg_num_history(e49 pg_nums {1={22=1,42=32}} deleted_pools ) 2020-03-11T22:41:48.557 INFO:tasks.workunit.client.0.smithi130.stdout:Importing pgid 1.0 2020-03-11T22:41:48.557 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:48.557 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:41:48.600 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:48.600 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:41:48.608 INFO:tasks.workunit.client.0.smithi130.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:48.608 INFO:tasks.workunit.client.0.smithi130.stdout:snapset 0=[]:{} 2020-03-11T22:41:48.616 INFO:tasks.workunit.client.0.smithi130.stdout:write_pg epoch 49 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/49 n=3 ec=22/22 lis/c=47/47 les/c/f=49/49/0 sis=47 pruub=13.302053139s) 2020-03-11T22:41:48.625 INFO:tasks.workunit.client.0.smithi130.stdout:Import successful 2020-03-11T22:41:49.144 INFO:tasks.workunit.client.0.smithi130.stdout:reviving divergent 1 2020-03-11T22:41:49.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:41:49.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:49.474 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:49.475 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.110563+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.475 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.107568+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.475 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.092166+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.475 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.106948+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.475 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:41:33.109439+0000 30'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.475 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108502+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.476 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.089047+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.476 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.091968+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.476 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.106137+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.476 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.109243+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.476 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.109570+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.476 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.089783+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.476 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.091673+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.477 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.090936+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.477 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:41:33.108864+0000 30'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.477 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.109769+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.477 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.106511+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.477 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.091392+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.477 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.089214+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.477 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108708+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.107816+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:41:33.109979+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.110310+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108260+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.088957+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:41:33.092512+0000 30'100 49:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.108181+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.479 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:41:33.109303+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.479 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:41:33.108502+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.479 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:41:33.090485+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.479 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:41:33.091079+0000 43'101 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.479 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:41:33.107236+0000 30'100 49:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:41:49.479 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:41:49.479 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:49.480 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:41:49.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:41:49.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:49.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:49.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:49.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:49.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:49.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:49.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:49.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:49.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:49.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:49.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:49.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:49.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:49.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:49.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:49.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:49.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:49.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:49.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:49.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:41:49.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:41:49.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:41:49.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:49.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:49.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:49.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:49.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:49.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:49.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:49.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:49.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:49.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:49.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:49.493 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:41:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:49.514 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:49.510+0000 7f51dc853ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:49.515 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:49.512+0000 7f51dc853ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:49.516 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:49.513+0000 7f51dc853ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:49.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:41:49.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:49.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:49.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:49.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:49.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:49.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:49.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:49.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:49.836 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:41:50.032 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:50.028+0000 7f51dc853ec0 -1 Falling back to public interface 2020-03-11T22:41:50.178 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42052,v1:127.0.0.1:6819/42052] [v2:127.0.0.1:6820/42052,v1:127.0.0.1:6821/42052] exists,up 53637eac-f654-4841-82d1-b00b8684e01b 2020-03-11T22:41:50.179 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:41:50.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:50.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:50.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:50.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:41:50.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:50.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:50.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:50.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:50.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:50.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:50.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:50.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:50.547 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42052,v1:127.0.0.1:6819/42052] [v2:127.0.0.1:6820/42052,v1:127.0.0.1:6821/42052] exists,up 53637eac-f654-4841-82d1-b00b8684e01b 2020-03-11T22:41:50.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:50.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:50.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:50.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:41:50.768 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:41:50.765+0000 7f51dc853ec0 -1 osd.1 49 log_to_monitors {default=true} 2020-03-11T22:42:10.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:42:10.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:42:10.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:42:10.550 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:10.550 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:10.550 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:10.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:42:10.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:10.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:42:10.648 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:42:10.648 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [], 2020-03-11T22:42:10.648 INFO:tasks.workunit.client.0.smithi130.stdout: "num_ops": 0 2020-03-11T22:42:10.648 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:42:10.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:42:10.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:10.656 INFO:tasks.workunit.client.0.smithi130.stdout:allowing recovery 2020-03-11T22:42:10.982 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:10.982 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:00.427423+0000 30'100 59:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.982 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:42:01.890256+0000 30'100 59:152 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:01.890211+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.983 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.400708+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.983 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:51.634131+0000 30'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.983 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:00.427319+0000 30'100 59:156 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.983 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 4 0 0 0 0 1932 0 0 16 16 active+clean 2020-03-11T22:42:00.863449+0000 30'100 59:152 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:00.863404+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.983 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:59.502075+0000 30'100 62:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:41:59.502028+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.983 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:51.633228+0000 30'100 59:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.983 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:00.426689+0000 30'100 59:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.984 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:58.458323+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.984 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.414447+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.984 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:57.794860+0000 30'100 59:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:57.794814+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.984 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:00.426800+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.984 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:58.458288+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.984 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.401562+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.984 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:51.634454+0000 30'100 59:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.985 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:56.400909+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.985 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:42:02.801556+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:02.801507+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.985 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:00.397094+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:41:57.461868+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.985 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:58.449587+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.985 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:51.631394+0000 30'100 59:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.985 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:56.414817+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:41:58.459225+0000 49'101 59:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:42:00.427008+0000 30'100 59:155 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:41:53.812041+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:58.813856+0000 30'100 59:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:58.813811+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:42:00.550820+0000 30'100 62:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:42:00.550757+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:58.459156+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.986 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:00.427083+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.987 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:42:04.774850+0000 30'100 59:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:04.774800+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.987 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.413682+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.987 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 4 0 0 0 0 1579 0 0 17 17 active+clean 2020-03-11T22:41:59.858857+0000 43'101 59:153 [0,1,2] 0 [0,1,2] 0 43'101 2020-03-11T22:41:59.858814+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.987 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:00.425997+0000 30'100 59:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:10.988 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:42:10.988 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:10.988 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:42:10.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:42:10.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:42:11.112 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:11.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:42:11.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:42:11.240 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:11.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:42:11.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:42:11.369 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:11.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:42:11.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:11.376 INFO:tasks.workunit.client.0.smithi130.stdout:reading divergent objects 2020-03-11T22:42:11.694 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:11.694 INFO:tasks.workunit.client.0.smithi130.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:00.427423+0000 30'100 59:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.694 INFO:tasks.workunit.client.0.smithi130.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:42:01.890256+0000 30'100 59:152 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:01.890211+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.694 INFO:tasks.workunit.client.0.smithi130.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.400708+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.695 INFO:tasks.workunit.client.0.smithi130.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:51.634131+0000 30'100 59:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.695 INFO:tasks.workunit.client.0.smithi130.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:00.427319+0000 30'100 59:156 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.695 INFO:tasks.workunit.client.0.smithi130.stdout:1.1a 4 0 0 0 0 1932 0 0 16 16 active+clean 2020-03-11T22:42:00.863449+0000 30'100 59:152 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:42:00.863404+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.695 INFO:tasks.workunit.client.0.smithi130.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:59.502075+0000 30'100 62:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:41:59.502028+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.695 INFO:tasks.workunit.client.0.smithi130.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:51.633228+0000 30'100 59:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.695 INFO:tasks.workunit.client.0.smithi130.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:00.426689+0000 30'100 59:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.695 INFO:tasks.workunit.client.0.smithi130.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:58.458323+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.696 INFO:tasks.workunit.client.0.smithi130.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.414447+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.696 INFO:tasks.workunit.client.0.smithi130.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:57.794860+0000 30'100 59:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:57.794814+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.696 INFO:tasks.workunit.client.0.smithi130.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:42:00.426800+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.696 INFO:tasks.workunit.client.0.smithi130.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:58.458288+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.696 INFO:tasks.workunit.client.0.smithi130.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.401562+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.696 INFO:tasks.workunit.client.0.smithi130.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:51.634454+0000 30'100 59:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.696 INFO:tasks.workunit.client.0.smithi130.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:56.400909+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.697 INFO:tasks.workunit.client.0.smithi130.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:42:02.801556+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:02.801507+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.697 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:00.397094+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:41:57.461868+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.697 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:58.449587+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.697 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:41:51.631394+0000 30'100 59:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.697 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:56.414817+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.697 INFO:tasks.workunit.client.0.smithi130.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:41:58.459225+0000 49'101 59:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.697 INFO:tasks.workunit.client.0.smithi130.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:42:00.427008+0000 30'100 59:155 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:41:53.812041+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.698 INFO:tasks.workunit.client.0.smithi130.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:58.813856+0000 30'100 59:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:41:58.813811+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.698 INFO:tasks.workunit.client.0.smithi130.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:42:00.550820+0000 30'100 62:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:42:00.550757+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.698 INFO:tasks.workunit.client.0.smithi130.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:41:58.459156+0000 30'100 59:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.698 INFO:tasks.workunit.client.0.smithi130.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:42:00.427083+0000 30'100 59:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.698 INFO:tasks.workunit.client.0.smithi130.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:42:04.774850+0000 30'100 59:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:42:04.774800+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.698 INFO:tasks.workunit.client.0.smithi130.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:56.413682+0000 30'100 62:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.699 INFO:tasks.workunit.client.0.smithi130.stdout:1.e 4 0 0 0 0 1579 0 0 17 17 active+clean 2020-03-11T22:41:59.858857+0000 43'101 59:153 [0,1,2] 0 [0,1,2] 0 43'101 2020-03-11T22:41:59.858814+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.699 INFO:tasks.workunit.client.0.smithi130.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:42:00.425997+0000 30'100 59:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:39:33.244374+0000 0'0 2020-03-11T22:39:33.244374+0000 0 2020-03-11T22:42:11.699 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:42:11.699 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:11.700 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:42:11.704 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:42:11.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:42:11.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:11.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:42:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:42:11.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:11.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:42:11.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:11.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:42:11.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:42:11.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:11.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:42:11.867 INFO:tasks.workunit.client.0.smithi130.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:42:11.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:11.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:42:11.931 INFO:tasks.workunit.client.0.smithi130.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:42:11.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:42:11.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:11.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:11.944 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4539.0:1 2020-03-11T22:39:56.122096+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.944 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 20 _merge_object_divergent_entries: keeping 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4539.0:1 2020-03-11T22:39:56.122096+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.944 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:11.945 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-03-11T22:42:11.945 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:42:11.945 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:11.945 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 30'106 2020-03-11T22:42:11.946 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4539.0:1 2020-03-11T22:39:56.122096+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.946 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:42:11.946 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'89 2020-03-11T22:42:11.946 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 30'105 (30'4) delete 1:70b3bf12:::existing_4:head by client.4542.0:1 2020-03-11T22:39:56.123590+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:11.947 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 20 _merge_object_divergent_entries: keeping 30'105 (30'4) delete 1:70b3bf12:::existing_4:head by client.4542.0:1 2020-03-11T22:39:56.123590+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:11.947 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:11.947 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 30'105 last_divergent_update: 30'105 2020-03-11T22:42:11.947 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: more recent entry found: 43'101 (30'4) modify 1:70b3bf12:::existing_4:head by client.4654.0:1 2020-03-11T22:40:49.946217+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:42:11.948 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 30'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4535.0:1 2020-03-11T22:39:56.117222+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.948 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 20 _merge_object_divergent_entries: keeping 30'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4535.0:1 2020-03-11T22:39:56.117222+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.948 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:11.948 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-03-11T22:42:11.949 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:42:11.949 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.703+0000 7f01b50c0700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:11.949 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:39:56.124748+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:11.949 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 20 _merge_object_divergent_entries: keeping 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:39:56.124748+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:11.950 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:42:11.950 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 30'106 last_divergent_update: 30'106 2020-03-11T22:42:11.950 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:42:11.950 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:11.950 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 30'106 2020-03-11T22:42:11.951 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:39:56.124748+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:11.951 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:42:11.951 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.722+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'84 2020-03-11T22:42:11.951 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4531.0:1 2020-03-11T22:39:56.107172+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.952 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 20 _merge_object_divergent_entries: keeping 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4531.0:1 2020-03-11T22:39:56.107172+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.952 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:11.953 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 30'102 last_divergent_update: 30'102 2020-03-11T22:42:11.953 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:42:11.953 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:11.954 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 30'106 2020-03-11T22:42:11.954 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4531.0:1 2020-03-11T22:39:56.107172+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.954 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:42:11.954 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'90 2020-03-11T22:42:11.954 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 30'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4529.0:1 2020-03-11T22:39:56.104150+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.955 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 20 _merge_object_divergent_entries: keeping 30'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4529.0:1 2020-03-11T22:39:56.104150+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:11.955 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:11.955 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 30'101 last_divergent_update: 30'101 2020-03-11T22:42:11.955 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:42:11.956 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:32.723+0000 7f01b60c2700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:11.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:11.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:11.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:11.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:11.968 INFO:tasks.workunit.client.0.smithi130.stdout:success 2020-03-11T22:42:11.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:42:11.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17424.out 2020-03-11T22:42:11.970 INFO:tasks.workunit.client.0.smithi130.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17424.out': No such file or directory 2020-03-11T22:42:11.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:42:11.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:42:11.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:42:12.634 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' does not exist 2020-03-11T22:42:12.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:42:12.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:12.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:12.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:12.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:42:12.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:12.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:12.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:12.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:12.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:12.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:12.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:12.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:12.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:12.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:12.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:12.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:12.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:12.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:12.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:12.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:12.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:12.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:12.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:12.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:12.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:12.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:12.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:12.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:12.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:12.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:42:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:42:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:42:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:42:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:42:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:12.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:12.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:12.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:12.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:12.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:12.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:12.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:12.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:12.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:12.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:12.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:12.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:12.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:12.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:12.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:12.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:12.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:12.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:12.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:12.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:42:12.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:12.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:12.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:42:12.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:42:12.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:12.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:12.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17424 2020-03-11T22:42:12.994 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:42:12.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:42:12.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:42:12.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:42:12.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:42:12.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:42:12.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:42:12.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:42:12.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:42:12.995 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:42:12.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:42:12.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:42:12.996 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:42:12.997 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:42:12.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:42:12.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:42:12.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:42:12.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:42:12.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:42:12.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:42:12.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:42:13.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:42:13.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:13.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:13.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:13.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:13.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:13.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:13.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:42:13.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:42:13.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:42:13.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:13.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:13.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:42:13.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:42:13.455 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:42:13.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:13.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get fsid 2020-03-11T22:42:13.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:42:13.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:13.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:13.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:42:13.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:42:13.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:13.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:13.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:13.545 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:13.545 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:13.545 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:13.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-mon.a.asok 2020-03-11T22:42:13.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:13.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17424/ceph-mon.a.asok config get mon_host 2020-03-11T22:42:13.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:42:13.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:42:13.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:42:13.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:42:13.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:42:13.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:42:13.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:42:14.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:14.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:14.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:14.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:42:14.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:14.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:14.581 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d34a39d8-760e-4854-8c86-11e878da47ec 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 d34a39d8-760e-4854-8c86-11e878da47ec' 2020-03-11T22:42:14.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:14.586 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 d34a39d8-760e-4854-8c86-11e878da47ec 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBGaWleP5MiIxAAccuCFcDfq/EjBWKF6tMpYQ== 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBGaWleP5MiIxAAccuCFcDfq/EjBWKF6tMpYQ=="}' 2020-03-11T22:42:14.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d34a39d8-760e-4854-8c86-11e878da47ec -i td/divergent-priors/0/new.json 2020-03-11T22:42:14.927 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:42:14.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:42:14.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBGaWleP5MiIxAAccuCFcDfq/EjBWKF6tMpYQ== --osd-uuid d34a39d8-760e-4854-8c86-11e878da47ec 2020-03-11T22:42:14.966 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:14.963+0000 7f15962f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:14.968 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:14.965+0000 7f15962f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:14.970 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:14.966+0000 7f15962f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:14.970 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:14.966+0000 7f15962f1ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:14.970 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:14.966+0000 7f15962f1ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:14.970 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:14.966+0000 7f15962f1ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:15.052 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:15.049+0000 7f15962f1ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:42:18.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:42:18.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:18.036 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:42:18.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:42:18.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:18.513 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:42:18.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:42:18.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:18.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:18.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:18.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:18.539 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:18.536+0000 7f69c7288ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:18.562 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:18.558+0000 7f69c7288ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:18.563 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:18.559+0000 7f69c7288ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:18.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:42:18.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:18.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:18.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:18.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:18.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:18.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:18.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:18.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:18.860 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:42:19.077 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:19.074+0000 7f69c7288ec0 -1 Falling back to public interface 2020-03-11T22:42:19.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:19.793 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:19.791+0000 7f69c7288ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:42:20.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:20.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:20.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:20.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:20.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:20.203 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:42:20.587 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43554,v1:127.0.0.1:6803/43554] [v2:127.0.0.1:6804/43554,v1:127.0.0.1:6805/43554] exists,up d34a39d8-760e-4854-8c86-11e878da47ec 2020-03-11T22:42:20.587 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5 2020-03-11T22:42:20.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:20.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:20.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:20.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:20.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:42:20.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:20.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:20.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:42:20.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:20.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:20.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:20.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:20.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:20.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:20.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:20.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:20.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:20.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:20.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:20.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:20.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:20.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:20.598 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:20.598 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:20.600 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:20.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:20.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:20.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:20.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:20.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:20.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:20.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:20.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:20.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:20.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:20.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:20.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:20.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:20.605 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:20.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5 2020-03-11T22:42:20.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5' 2020-03-11T22:42:20.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:20.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBMaWleFlfGIxAA2yKDvyCEcvy7P4kyxy66iA== 2020-03-11T22:42:20.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBMaWleFlfGIxAA2yKDvyCEcvy7P4kyxy66iA=="}' 2020-03-11T22:42:20.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5 -i td/divergent-priors/1/new.json 2020-03-11T22:42:21.072 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:42:21.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:42:21.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBMaWleFlfGIxAA2yKDvyCEcvy7P4kyxy66iA== --osd-uuid 05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5 2020-03-11T22:42:21.110 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:21.108+0000 7f99a28c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:21.116 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:21.114+0000 7f99a28c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:21.117 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:21.115+0000 7f99a28c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:21.117 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:21.115+0000 7f99a28c2ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:21.117 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:21.115+0000 7f99a28c2ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:21.117 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:21.115+0000 7f99a28c2ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:21.216 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:21.214+0000 7f99a28c2ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:42:24.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:42:24.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:24.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:42:24.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:24.235 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:42:24.691 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:42:24.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:42:24.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:24.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:24.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:24.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:24.718 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:24.716+0000 7fd466226ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:24.720 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:24.718+0000 7fd466226ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:24.722 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:24.720+0000 7fd466226ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:25.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:42:25.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:25.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:25.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:25.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:25.042 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:42:25.240 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:25.238+0000 7fd466226ec0 -1 Falling back to public interface 2020-03-11T22:42:25.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:26.031 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:26.029+0000 7fd466226ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:42:26.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:26.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:26.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:26.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:26.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:26.419 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:42:26.777 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43909,v1:127.0.0.1:6811/43909] [v2:127.0.0.1:6812/43909,v1:127.0.0.1:6813/43909] exists,up 05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5 2020-03-11T22:42:26.778 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 9bd7863a-9ae2-4649-8393-d7b70b4864ee 2020-03-11T22:42:26.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:26.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:26.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:26.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:26.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:42:26.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:42:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:42:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:42:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:42:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:26.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:26.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:26.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:26.781 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:26.781 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:26.781 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:42:26.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:26.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:42:26.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:26.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:26.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:26.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:26.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:26.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:26.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:26.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:26.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:26.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:26.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:42:26.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:26.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9bd7863a-9ae2-4649-8393-d7b70b4864ee 2020-03-11T22:42:26.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 9bd7863a-9ae2-4649-8393-d7b70b4864ee' 2020-03-11T22:42:26.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:26.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBSaWleeu1GLxAAbtPgw/iMuZgJJzLuuZ1XCg== 2020-03-11T22:42:26.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBSaWleeu1GLxAAbtPgw/iMuZgJJzLuuZ1XCg=="}' 2020-03-11T22:42:26.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9bd7863a-9ae2-4649-8393-d7b70b4864ee -i td/divergent-priors/2/new.json 2020-03-11T22:42:27.295 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:42:27.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:42:27.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBSaWleeu1GLxAAbtPgw/iMuZgJJzLuuZ1XCg== --osd-uuid 9bd7863a-9ae2-4649-8393-d7b70b4864ee 2020-03-11T22:42:27.334 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:27.331+0000 7f2fd405fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:27.335 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:27.333+0000 7f2fd405fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:27.336 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:27.334+0000 7f2fd405fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:27.337 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:27.334+0000 7f2fd405fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:27.337 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:27.334+0000 7f2fd405fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:27.334+0000 7f2fd405fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:27.414 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:27.410+0000 7f2fd405fec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:42:30.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:42:30.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:30.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:42:30.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:30.499 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:42:30.961 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:42:30.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:42:30.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:30.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:30.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:30.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:30.986 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:30.984+0000 7f20a2f68ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:30.988 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:30.986+0000 7f20a2f68ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:30.989 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:30.987+0000 7f20a2f68ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:31.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:42:31.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:31.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:31.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:31.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:31.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:31.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:31.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:31.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:31.308 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:42:31.505 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:31.503+0000 7f20a2f68ec0 -1 Falling back to public interface 2020-03-11T22:42:31.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:32.228 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:42:32.226+0000 7f20a2f68ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:42:32.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:32.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:32.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:32.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:32.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:32.659 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:42:33.035 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44264,v1:127.0.0.1:6819/44264] [v2:127.0.0.1:6820/44264,v1:127.0.0.1:6821/44264] exists,up 9bd7863a-9ae2-4649-8393-d7b70b4864ee 2020-03-11T22:42:33.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:33.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:33.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:33.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:42:33.737 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2020-03-11T22:42:33.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:42:34.365 INFO:tasks.workunit.client.0.smithi130.stderr:noin is set 2020-03-11T22:42:34.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:42:34.983 INFO:tasks.workunit.client.0.smithi130.stderr:nodown is set 2020-03-11T22:42:34.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:42:34.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:42:34.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:42:34.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:42:34.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:42:34.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:42:35.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:42:35.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:42:36.369 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2020-03-11T22:42:36.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:42:37.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:42:37.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:42:38.004 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:42:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:42:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:38.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:42:38.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:38.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:38.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:38.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:38.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:38.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:38.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:42:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:42:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:38.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:38.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:42:38.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:42:38.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:38.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-03-11T22:42:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2020-03-11T22:42:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:38.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:38.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=55834574853 2020-03-11T22:42:38.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-55834574853' 2020-03-11T22:42:38.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:38.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:42:38.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:38.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:42:38.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:38.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:42:38.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:42:38.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:38.855 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:42:39.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-11T22:42:39.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:40.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:40.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:40.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:42:40.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:40.554 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705669 2020-03-11T22:42:40.554 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:40.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:40.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705669 2020-03-11T22:42:40.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:40.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705669 2020-03-11T22:42:40.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-03-11T22:42:40.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:40.557 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 38654705669 2020-03-11T22:42:40.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705669 -lt 38654705669 2020-03-11T22:42:40.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:40.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-55834574853 2020-03-11T22:42:40.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:40.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:40.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-55834574853 2020-03-11T22:42:40.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:40.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=55834574853 2020-03-11T22:42:40.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 55834574853' 2020-03-11T22:42:40.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:40.927 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 55834574853 2020-03-11T22:42:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 55834574853 -lt 55834574853 2020-03-11T22:42:41.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:41.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:41.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:41.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:41.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:41.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:41.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:41.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:41.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:41.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:41.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:42.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:42.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:42.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:42.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:42:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:42:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:42.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:42.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:42.838 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:42:42.838 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:42:42.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:42.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:42.838 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:42.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:42:42.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:42:42.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:42.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:43.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705672 2020-03-11T22:42:43.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705672' 2020-03-11T22:42:43.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:43.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:43.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=55834574855 2020-03-11T22:42:43.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-55834574855' 2020-03-11T22:42:43.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:43.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:42:43.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:43.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:43.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:42:43.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:43.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:42:43.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:42:43.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:43.243 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:42:43.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:42:43.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:44.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:44.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:44.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:42:44.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:44.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705672 2020-03-11T22:42:44.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:44.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:44.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705672 2020-03-11T22:42:44.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:44.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705672 2020-03-11T22:42:44.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2020-03-11T22:42:44.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:44.939 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 38654705672 2020-03-11T22:42:45.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705672 -lt 38654705672 2020-03-11T22:42:45.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:45.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-55834574855 2020-03-11T22:42:45.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:45.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:45.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-55834574855 2020-03-11T22:42:45.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:45.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=55834574855 2020-03-11T22:42:45.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2020-03-11T22:42:45.284 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 55834574855 2020-03-11T22:42:45.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:45.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 55834574856 -lt 55834574855 2020-03-11T22:42:45.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:42:45.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:45.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:45.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:45.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:45.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:42:45.629 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:45.629 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:45.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:45.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:45.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:45.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:45.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:45.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:45.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:45.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:46.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:46.065 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:42:46.065 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:42:46.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:46.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:46.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:46.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:42:46.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:42:46.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:46.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:46.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705674 2020-03-11T22:42:46.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705674' 2020-03-11T22:42:46.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:46.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:46.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=55834574858 2020-03-11T22:42:46.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705674 2-55834574858' 2020-03-11T22:42:46.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:46.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:42:46.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:46.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:46.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:42:46.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:46.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:42:46.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:42:46.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:46.479 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:42:46.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836493 2020-03-11T22:42:46.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:47.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:47.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:48.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:42:48.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:48.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705674 2020-03-11T22:42:48.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:48.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:48.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705674 2020-03-11T22:42:48.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:48.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705674 2020-03-11T22:42:48.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2020-03-11T22:42:48.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:48.180 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 38654705674 2020-03-11T22:42:48.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705675 -lt 38654705674 2020-03-11T22:42:48.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:48.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-55834574858 2020-03-11T22:42:48.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:48.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:48.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-55834574858 2020-03-11T22:42:48.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:48.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=55834574858 2020-03-11T22:42:48.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2020-03-11T22:42:48.548 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 55834574858 2020-03-11T22:42:48.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:48.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 55834574858 -lt 55834574858 2020-03-11T22:42:48.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:48.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:48.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:49.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:49.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:49.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:49.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:49.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:49.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:49.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:49.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:49.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:49.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:49.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:50.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:50.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:50.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:50.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:42:50.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:42:50.442 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:42:50.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:42:50.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:42:50.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:42:50.451 INFO:tasks.workunit.client.0.smithi130.stdout:primary and soon to be divergent is 1 2020-03-11T22:42:50.769 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:50.769 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:42:36.999708+0000 0'0 25:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:42:50.769 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:42:50.770 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:50.771 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:42:50.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:42:50.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:50.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:42:50.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:42:50.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:50.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:42:50.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:42:50.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:50.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:42:50.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:42:50.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:42:50.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:42:50.781 INFO:tasks.workunit.client.0.smithi130.stdout:writing initial objects 2020-03-11T22:42:50.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:42:50.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:42:50.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:42:50.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:42:50.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:42:50.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:42:51.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:42:51.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:42:51.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:42:51.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:42:51.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:42:51.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:42:51.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:42:51.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:42:51.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:42:51.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:42:51.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:42:51.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:42:51.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:42:51.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:42:52.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:42:52.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:42:52.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:42:52.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:42:52.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:42:52.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:42:52.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:42:52.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:42:52.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:42:52.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:42:52.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:42:52.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:42:52.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:42:52.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:42:52.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:42:52.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:42:52.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:42:52.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:42:52.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:42:53.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:42:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:42:53.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:42:53.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:42:53.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:42:53.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:42:53.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:42:53.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:42:53.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:42:53.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:42:53.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:42:53.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:42:53.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:42:53.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:42:53.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:42:53.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:42:54.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:42:54.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:42:54.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:42:54.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:42:54.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:42:54.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:42:54.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:42:54.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:42:54.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:42:54.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:42:54.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:42:54.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:42:54.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:42:54.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:42:54.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:42:54.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:42:55.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:42:55.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:42:55.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:42:55.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:42:55.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:42:55.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:42:55.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:42:55.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:42:55.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:42:55.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:42:55.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:42:55.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:42:55.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:42:55.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:42:55.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:42:55.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:42:56.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:42:56.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:42:56.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:42:56.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:42:56.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:42:56.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:42:56.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:42:56.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:42:56.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:42:56.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:42:56.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:42:56.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:42:56.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:42:56.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:42:56.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:42:56.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:42:56.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:56.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:56.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:56.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:56.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:42:56.711 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:56.711 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:56.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:56.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:56.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:56.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:56.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:56.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:56.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:56.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:57.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:57.111 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:42:57.112 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:42:57.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:57.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:57.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:57.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836497 2020-03-11T22:42:57.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836497' 2020-03-11T22:42:57.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:57.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:57.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705678 2020-03-11T22:42:57.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836497 1-38654705678' 2020-03-11T22:42:57.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:57.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:57.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=55834574862 2020-03-11T22:42:57.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836497 1-38654705678 2-55834574862' 2020-03-11T22:42:57.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:57.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836497 2020-03-11T22:42:57.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:57.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:57.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836497 2020-03-11T22:42:57.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:57.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836497 2020-03-11T22:42:57.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2020-03-11T22:42:57.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:57.501 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836497 2020-03-11T22:42:57.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836497 2020-03-11T22:42:57.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:58.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:58.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:59.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836497 2020-03-11T22:42:59.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:59.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705678 2020-03-11T22:42:59.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:59.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:59.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705678 2020-03-11T22:42:59.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:59.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705678 2020-03-11T22:42:59.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705678' 2020-03-11T22:42:59.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:59.197 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 38654705678 2020-03-11T22:42:59.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 38654705678 2020-03-11T22:42:59.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:59.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-55834574862 2020-03-11T22:42:59.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:59.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:59.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-55834574862 2020-03-11T22:42:59.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:59.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=55834574862 2020-03-11T22:42:59.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 55834574862' 2020-03-11T22:42:59.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:59.546 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 55834574862 2020-03-11T22:42:59.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 55834574862 -lt 55834574862 2020-03-11T22:42:59.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:59.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:59.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:00.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:00.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:00.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:00.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:00.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:00.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:00.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:00.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:00.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:43:00.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:00.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:00.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:01.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:43:01.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:43:01.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:43:01.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:43:01.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:43:01.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:43:01.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:43:01.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:43:01.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:43:01.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:43:01.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:01.509 INFO:tasks.workunit.client.0.smithi130.stdout:blackholing osds 0 2 2020-03-11T22:43:01.829 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:01.829 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:42:36.999708+0000 27'100 27:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:43:01.829 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:43:01.829 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:01.830 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:43:01.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:01.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:43:01.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:43:01.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:43:01.840 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:01.840 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:01.840 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:43:01.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:43:01.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:43:01.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:43:01.926 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:43:01.927 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:01.927 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:43:01.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:01.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:43:01.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:43:01.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:43:01.935 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:01.935 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:01.935 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:43:01.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:43:01.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:43:01.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:43:02.021 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:43:02.022 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:02.022 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:43:02.029 INFO:tasks.workunit.client.0.smithi130.stdout:writing divergent object 2020-03-11T22:43:02.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:43:02.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:43:02.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:43:03.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:43:03.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:43:03.187 INFO:tasks.workunit.client.0.smithi130.stdout:created pool test snap snap1 2020-03-11T22:43:03.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:43:03.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:43:03.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:43:13.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:43:13.195 INFO:tasks.workunit.client.0.smithi130.stderr:rados: no process found 2020-03-11T22:43:13.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:43:13.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:13.195 INFO:tasks.workunit.client.0.smithi130.stdout:killing all the osds 2020-03-11T22:43:13.524 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:13.524 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:42:36.999708+0000 28'104 28:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:43:13.524 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:43:13.524 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:13.526 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:43:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:43:13.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:13.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:13.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:13.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:13.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:13.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:43:14.123 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2020-03-11T22:43:14.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:14.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:43:14.608 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already down. 2020-03-11T22:43:14.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:14.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:43:15.099 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already down. 2020-03-11T22:43:15.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:15.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:43:15.575 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2020-03-11T22:43:15.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:15.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:43:16.090 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2020-03-11T22:43:16.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:43:16.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:16.098 INFO:tasks.workunit.client.0.smithi130.stdout:bringing up non_divergent 0 2 2020-03-11T22:43:16.422 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:16.422 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:42:36.999708+0000 28'104 28:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:43:16.422 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:43:16.423 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:16.424 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:43:16.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:16.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:43:16.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:16.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:16.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:43:16.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:16.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:16.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:16.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:16.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:16.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:16.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:16.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:16.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:16.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:43:16.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:43:16.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:43:16.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:16.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:16.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:16.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:43:16.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:16.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:43:16.441 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:43:16.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:43:16.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:16.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:16.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:16.461 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:16.459+0000 7fbb09ed1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:16.463 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:16.461+0000 7fbb09ed1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:16.465 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:16.462+0000 7fbb09ed1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:16.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:43:16.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:16.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:16.785 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:43:16.986 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:16.984+0000 7fbb09ed1ec0 -1 Falling back to public interface 2020-03-11T22:43:17.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:17.847 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:17.845+0000 7fbb09ed1ec0 -1 osd.0 27 log_to_monitors {default=true} 2020-03-11T22:43:18.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:18.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:18.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:18.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:18.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:18.130 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:43:18.488 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/48667,v1:127.0.0.1:6803/48667] [v2:127.0.0.1:6804/48667,v1:127.0.0.1:6805/48667] exists,up d34a39d8-760e-4854-8c86-11e878da47ec 2020-03-11T22:43:18.488 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:43:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:43:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:43:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:18.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:18.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:18.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:18.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:18.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:18.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:18.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:18.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:43:18.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:43:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:43:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:18.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:18.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:43:18.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:18.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:43:18.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:43:18.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:18.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:18.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:18.514 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:18.511+0000 7f21e00e8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:18.515 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:18.513+0000 7f21e00e8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:18.516 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:18.514+0000 7f21e00e8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:18.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:43:18.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:18.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:18.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:18.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:18.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:18.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:18.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:18.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:18.881 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:43:19.032 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:19.029+0000 7f21e00e8ec0 -1 Falling back to public interface 2020-03-11T22:43:19.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:19.895 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:43:19.893+0000 7f21e00e8ec0 -1 osd.2 27 log_to_monitors {default=true} 2020-03-11T22:43:20.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:20.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:20.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:20.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:20.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:20.226 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:43:20.573 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [13,27) [v2:127.0.0.1:6810/48861,v1:127.0.0.1:6811/48861] [v2:127.0.0.1:6812/48861,v1:127.0.0.1:6813/48861] exists,up 9bd7863a-9ae2-4649-8393-d7b70b4864ee 2020-03-11T22:43:20.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:20.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:20.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:20.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:20.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:43:21.065 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already in. 2020-03-11T22:43:21.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:21.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:43:21.735 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2020-03-11T22:43:21.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:43:26.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:43:26.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:43:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:43:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:26.746 INFO:tasks.workunit.client.0.smithi130.stdout:writing non-divergent object existing_4 2020-03-11T22:43:27.075 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:27.075 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded+wait 2020-03-11T22:43:21.788136+0000 27'100 40:129 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:43:27.075 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:43:27.075 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:43:27.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:43:37.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:43:37.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:43:37.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:37.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:37.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:37.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:37.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:43:37.511 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:37.512 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:37.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:37.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:37.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:37.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:37.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:37.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:37.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:37.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:37.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:37.924 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:43:37.924 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:43:37.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:37.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:37.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:38.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888071 2020-03-11T22:43:38.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888071' 2020-03-11T22:43:38.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:38.054 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:38.146 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:43:38.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:43:38.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888071 1-' 2020-03-11T22:43:38.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:38.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:38.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855366 2020-03-11T22:43:38.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888071 1- 2-150323855366' 2020-03-11T22:43:38.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:38.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888071 2020-03-11T22:43:38.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:38.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:38.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888071 2020-03-11T22:43:38.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:38.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888071 2020-03-11T22:43:38.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888071' 2020-03-11T22:43:38.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:38.287 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 146028888071 2020-03-11T22:43:38.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888071 2020-03-11T22:43:38.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:39.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:39.627 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:40.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888071 -lt 146028888071 2020-03-11T22:43:40.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:40.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:43:40.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:40.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:40.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:43:40.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:40.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:43:40.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:43:40.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:40.020 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 2020-03-11T22:43:40.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705682 -lt 2020-03-11T22:43:40.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705682: unary operator expected 2020-03-11T22:43:40.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:40.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855366 2020-03-11T22:43:40.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:40.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:40.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855366 2020-03-11T22:43:40.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:40.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855366 2020-03-11T22:43:40.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855366' 2020-03-11T22:43:40.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:40.392 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 150323855366 2020-03-11T22:43:40.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855366 -lt 150323855366 2020-03-11T22:43:40.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:40.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:40.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:41.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:41.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:41.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:41.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:41.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:41.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:41.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:41.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:41.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:41.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:41.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:41.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:41.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:41.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:43:41.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:43:41.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:43:41.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:41.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:43:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:42.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:42.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:42.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:42.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:42.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:42.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:42.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:42.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:42.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:42.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:42.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:42.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:42.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:42.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:42.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:42.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:42.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:42.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:42.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:43.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:43.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:43.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:43:43.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:43.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:43.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:43.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:43.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:43.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:44.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:44.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:44.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:44.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:44.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:44.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:44.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:44.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:44.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:44.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:44.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:44.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:43:44.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:44.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:43:45.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:45.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:45.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:45.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:45.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:45.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:45.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:45.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:45.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:45.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:45.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:45.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:45.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:45.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:45.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:45.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:45.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:45.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:45.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:45.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:45.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:46.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:46.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:46.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:43:46.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:46.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:43:47.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:47.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:47.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:47.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:47.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:47.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:47.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:47.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:47.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:47.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:47.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:47.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:47.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:47.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:47.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:47.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:47.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:47.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:47.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:47.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:47.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:48.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:48.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:48.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:43:48.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:48.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:43:49.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:49.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:49.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:49.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:49.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:49.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:49.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:49.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:50.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:50.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:50.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:50.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:50.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:50.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:51.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:51.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:51.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:43:51.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:51.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:43:54.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:54.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:54.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:54.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:54.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:54.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:54.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:54.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:54.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:54.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:54.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:55.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:55.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:55.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:43:55.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:55.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:44:02.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:02.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:02.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:02.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:02.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:02.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:02.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:02.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:02.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:02.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:02.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:02.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:02.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:02.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:02.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:02.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:02.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:02.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:02.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:02.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:02.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:03.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:03.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:03.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:44:03.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:03.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:44:10.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:10.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:10.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:10.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:10.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:10.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:10.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:11.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:11.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:11.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:11.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:11.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:11.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:11.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:11.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:11.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:11.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:11.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:44:11.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:44:12.233 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:44:12.233 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_fingerprint": "63286032-48e7-4f61-99d7-1878700996a8", 2020-03-11T22:44:12.233 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:44:12.233 INFO:tasks.workunit.client.0.smithi130.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:44:12.233 INFO:tasks.workunit.client.0.smithi130.stdout: "timestamp": "2020-03-11T22:44:12.229715+0000", 2020-03-11T22:44:12.233 INFO:tasks.workunit.client.0.smithi130.stdout: "tag": "", 2020-03-11T22:44:12.234 INFO:tasks.workunit.client.0.smithi130.stdout: "health": { 2020-03-11T22:44:12.234 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "HEALTH_WARN", 2020-03-11T22:44:12.234 INFO:tasks.workunit.client.0.smithi130.stdout: "checks": { 2020-03-11T22:44:12.234 INFO:tasks.workunit.client.0.smithi130.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:44:12.234 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:12.234 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:12.234 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 17 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [], 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: "OSD_DOWN": { 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:12.235 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 osds down", 2020-03-11T22:44:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:44:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:44:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "osd.1 (root=default,host=smithi130) is down" 2020-03-11T22:44:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: "PG_DEGRADED": { 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:44:12.237 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:44:12.238 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.238 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:44:12.238 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.238 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:44:12.238 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.238 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.238 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:12.239 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.239 INFO:tasks.workunit.client.0.smithi130.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:44:12.239 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:12.239 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:12.239 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:44:12.239 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:44:12.240 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.241 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.241 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:12.241 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.241 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.241 INFO:tasks.workunit.client.0.smithi130.stdout: "mutes": [] 2020-03-11T22:44:12.241 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.241 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_first_committed": 1, 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_last_committed": 1, 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap": { 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:42:13.015911Z", 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:42:13.015911Z", 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release": 15, 2020-03-11T22:44:12.242 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "features": { 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "persistent": [ 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "kraken", 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "luminous", 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "mimic", 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap-prune", 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "nautilus", 2020-03-11T22:44:12.243 INFO:tasks.workunit.client.0.smithi130.stdout: "octopus" 2020-03-11T22:44:12.244 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.244 INFO:tasks.workunit.client.0.smithi130.stdout: "optional": [] 2020-03-11T22:44:12.244 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.244 INFO:tasks.workunit.client.0.smithi130.stdout: "mons": [ 2020-03-11T22:44:12.244 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.244 INFO:tasks.workunit.client.0.smithi130.stdout: "rank": 0, 2020-03-11T22:44:12.244 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "a", 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 0 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.245 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.246 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.246 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:44:12.246 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:44:12.246 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 0, 2020-03-11T22:44:12.246 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 0 2020-03-11T22:44:12.246 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.246 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: "quorum": [ 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: 0 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap": { 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 40, 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:44:12.247 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:42:13.715168+0000", 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:43:21.654492+0000", 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "last_up_change": "2020-03-11T22:43:19.984849+0000", 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "last_in_change": "2020-03-11T22:43:21.454158+0000", 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_num": 5800384, 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_set": [ 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "nodown", 2020-03-11T22:44:12.248 INFO:tasks.workunit.client.0.smithi130.stdout: "noin", 2020-03-11T22:44:12.249 INFO:tasks.workunit.client.0.smithi130.stdout: "noout", 2020-03-11T22:44:12.249 INFO:tasks.workunit.client.0.smithi130.stdout: "pglog_hardlimit", 2020-03-11T22:44:12.249 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snapdirs", 2020-03-11T22:44:12.249 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_deletes", 2020-03-11T22:44:12.249 INFO:tasks.workunit.client.0.smithi130.stdout: "sortbitwise" 2020-03-11T22:44:12.249 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.249 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_version": 8, 2020-03-11T22:44:12.250 INFO:tasks.workunit.client.0.smithi130.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:44:12.250 INFO:tasks.workunit.client.0.smithi130.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:44:12.250 INFO:tasks.workunit.client.0.smithi130.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:44:12.250 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_snapshot": "", 2020-03-11T22:44:12.250 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_max": 1, 2020-03-11T22:44:12.250 INFO:tasks.workunit.client.0.smithi130.stdout: "max_osd": 3, 2020-03-11T22:44:12.250 INFO:tasks.workunit.client.0.smithi130.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: "min_compat_client": "jewel", 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: "require_osd_release": "octopus", 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: "pools": [ 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_name": "test", 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: "create_time": "2020-03-11T22:42:35.958624+0000", 2020-03-11T22:44:12.251 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": 16389, 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 3, 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "size": 3, 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 2, 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_rule": 1, 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "object_hash": 2, 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:44:12.252 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num": 1, 2020-03-11T22:44:12.253 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num": 1, 2020-03-11T22:44:12.253 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num_target": 1, 2020-03-11T22:44:12.253 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_target": 1, 2020-03-11T22:44:12.253 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_pending": 1, 2020-03-11T22:44:12.253 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pg_merge_meta": { 2020-03-11T22:44:12.253 INFO:tasks.workunit.client.0.smithi130.stdout: "source_pgid": "0.0", 2020-03-11T22:44:12.253 INFO:tasks.workunit.client.0.smithi130.stdout: "ready_epoch": 0, 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 0, 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 0, 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: "source_version": "0'0", 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: "target_version": "0'0" 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "28", 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend": "0", 2020-03-11T22:44:12.254 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:44:12.255 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:44:12.255 INFO:tasks.workunit.client.0.smithi130.stdout: "auid": 0, 2020-03-11T22:44:12.255 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_mode": "pool", 2020-03-11T22:44:12.255 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_seq": 1, 2020-03-11T22:44:12.255 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_epoch": 28, 2020-03-11T22:44:12.255 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_snaps": [ 2020-03-11T22:44:12.255 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: "snapid": 1, 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: "stamp": "2020-03-11T22:43:03.056898+0000", 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "snap1" 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps": "[]", 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_bytes": 0, 2020-03-11T22:44:12.256 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_objects": 0, 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "tiers": [], 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "tier_of": -1, 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "read_tier": -1, 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "write_tier": -1, 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_mode": "none", 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_bytes": 0, 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_objects": 0, 2020-03-11T22:44:12.257 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:44:12.258 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:44:12.258 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:44:12.258 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_flush_age": 0, 2020-03-11T22:44:12.258 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_evict_age": 0, 2020-03-11T22:44:12.258 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:44:12.258 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_params": { 2020-03-11T22:44:12.258 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "none" 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_period": 0, 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_count": 0, 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: "use_gmt_hitset": true, 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:44:12.259 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: "grade_table": [], 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: "stripe_width": 8192, 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: "expected_num_objects": 0, 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: "fast_read": false, 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: "options": {}, 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: "application_metadata": {} 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.260 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.261 INFO:tasks.workunit.client.0.smithi130.stdout: "osds": [ 2020-03-11T22:44:12.261 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.261 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:12.261 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "d34a39d8-760e-4854-8c86-11e878da47ec", 2020-03-11T22:44:12.261 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:44:12.261 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:44:12.261 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 5, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 27, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 34, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 39, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 29, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:44:12.262 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.263 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.264 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:44:12.264 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.264 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.264 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.264 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.265 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.266 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.266 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:44:12.266 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.266 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.266 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.266 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.266 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.267 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.268 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:44:12.268 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.268 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.268 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.268 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.268 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:12.268 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.269 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:44:12.270 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48667 2020-03-11T22:44:12.270 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.270 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.270 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.270 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6803/48667", 2020-03-11T22:44:12.270 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6805/48667", 2020-03-11T22:44:12.270 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6809/48667", 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6807/48667", 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.271 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:12.272 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5", 2020-03-11T22:44:12.272 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 0, 2020-03-11T22:44:12.272 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:44:12.272 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:44:12.272 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:44:12.272 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 0, 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 0, 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 9, 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 23, 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 30, 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.273 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.274 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.274 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:12.274 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.274 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.274 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.274 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.274 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.275 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.276 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:12.276 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.276 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.276 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.276 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.276 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:12.276 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.277 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.277 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.277 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.277 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:12.277 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.277 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.277 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.278 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:12.278 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.278 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.278 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.278 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.279 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:12.279 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.279 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.279 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.279 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.279 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:12.279 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.280 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:12.281 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:12.281 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.281 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.281 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.281 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/43909", 2020-03-11T22:44:12.281 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/43909", 2020-03-11T22:44:12.281 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43909", 2020-03-11T22:44:12.282 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43909", 2020-03-11T22:44:12.282 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:44:12.282 INFO:tasks.workunit.client.0.smithi130.stdout: "exists" 2020-03-11T22:44:12.282 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.282 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.282 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.282 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:12.283 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "9bd7863a-9ae2-4649-8393-d7b70b4864ee", 2020-03-11T22:44:12.283 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:44:12.283 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:44:12.283 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:44:12.283 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:44:12.283 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 13, 2020-03-11T22:44:12.283 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 27, 2020-03-11T22:44:12.284 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 35, 2020-03-11T22:44:12.284 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 0, 2020-03-11T22:44:12.284 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 31, 2020-03-11T22:44:12.284 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:44:12.284 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:12.284 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.284 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.285 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.285 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:12.285 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.285 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.285 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.285 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.285 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:12.286 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.286 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.286 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.286 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.286 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:44:12.286 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.286 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:12.287 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.288 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:12.289 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.289 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.289 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.289 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.289 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:12.289 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.289 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:12.290 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 48861 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.291 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/48861", 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/48861", 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48861", 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48861", 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:44:12.292 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.293 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.293 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.293 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_xinfo": [ 2020-03-11T22:44:12.293 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.293 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:12.293 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:43:13.926616+0000", 2020-03-11T22:44:12.293 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:20.806050+0000", 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.294 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:12.295 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:43:14.428449+0000", 2020-03-11T22:44:12.295 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:44:12.295 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:44:12.295 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:44:12.295 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:44:12.295 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:27.028030+0000", 2020-03-11T22:44:12.295 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:43:14.911639+0000", 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:44:12.296 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:44:12.297 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:33.196975+0000", 2020-03-11T22:44:12.297 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:44:12.297 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.297 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.297 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap": [], 2020-03-11T22:44:12.297 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap_items": [], 2020-03-11T22:44:12.297 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_temp": [], 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_temp": [], 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "blacklist": {}, 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profiles": { 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "default": { 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "2", 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:44:12.298 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van" 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: "myprofile": { 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: "crush-device-class": "", 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: "crush-failure-domain": "osd", 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: "crush-root": "default", 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:44:12.299 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "1", 2020-03-11T22:44:12.300 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:44:12.300 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van", 2020-03-11T22:44:12.300 INFO:tasks.workunit.client.0.smithi130.stdout: "w": "8" 2020-03-11T22:44:12.300 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.300 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.300 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps_queue": [], 2020-03-11T22:44:12.300 INFO:tasks.workunit.client.0.smithi130.stdout: "new_removed_snaps": [], 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: "new_purged_snaps": [], 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_node_flags": {}, 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: "device_class_flags": {} 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_metadata": [ 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:12.301 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:44:12.302 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6804/48667,v1:127.0.0.1:6805/48667]", 2020-03-11T22:44:12.302 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:44:12.302 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:44:12.302 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:12.302 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:12.302 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:12.302 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:12.303 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:12.304 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:44:12.304 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:12.304 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:12.304 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:12.304 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:44:12.304 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:44:12.305 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:12.305 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:44:12.305 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:44:12.305 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:12.305 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:44:12.305 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6802/48667,v1:127.0.0.1:6803/48667]", 2020-03-11T22:44:12.305 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/48667,v1:127.0.0.1:6809/48667]", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/48667,v1:127.0.0.1:6807/48667]", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:12.306 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:44:12.307 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:12.307 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:12.307 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:44:12.307 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:44:12.307 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:12.307 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:44:12.307 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.308 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.308 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:12.308 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:44:12.308 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/43909,v1:127.0.0.1:6813/43909]", 2020-03-11T22:44:12.308 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:44:12.308 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:44:12.308 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:12.309 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:12.309 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:12.309 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:12.309 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:12.309 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:12.309 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:12.309 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:44:12.310 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:12.310 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:12.310 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:12.310 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:12.310 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:44:12.310 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:12.310 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:12.311 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:44:12.312 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/43909,v1:127.0.0.1:6811/43909]", 2020-03-11T22:44:12.312 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:44:12.312 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43909,v1:127.0.0.1:6817/43909]", 2020-03-11T22:44:12.312 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43909,v1:127.0.0.1:6815/43909]", 2020-03-11T22:44:12.312 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:44:12.312 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:44:12.312 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:12.313 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:12.313 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:12.313 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:44:12.313 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:12.313 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:12.313 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:44:12.313 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/48861,v1:127.0.0.1:6813/48861]", 2020-03-11T22:44:12.314 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:12.315 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:12.316 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:44:12.316 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:12.316 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:12.316 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:12.316 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:12.316 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:44:12.316 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:12.317 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:12.317 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:12.317 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:44:12.317 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:44:12.317 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:12.317 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:44:12.317 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/48861,v1:127.0.0.1:6811/48861]", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48861,v1:127.0.0.1:6817/48861]", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48861,v1:127.0.0.1:6815/48861]", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:44:12.318 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:44:12.319 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:44:12.320 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:12.320 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:44:12.320 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.320 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.320 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_first_committed": 1, 2020-03-11T22:44:12.320 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_last_committed": 40, 2020-03-11T22:44:12.320 INFO:tasks.workunit.client.0.smithi130.stdout: "crushmap": { 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": [ 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.0", 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.321 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:12.322 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.1", 2020-03-11T22:44:12.322 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:44:12.322 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.322 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.322 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:12.322 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.2", 2020-03-11T22:44:12.322 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: "types": [ 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 0, 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd" 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.323 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.324 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:44:12.324 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "host" 2020-03-11T22:44:12.324 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.324 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.324 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 2, 2020-03-11T22:44:12.324 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "chassis" 2020-03-11T22:44:12.324 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 3, 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "rack" 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 4, 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "row" 2020-03-11T22:44:12.325 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.326 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.326 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 5, 2020-03-11T22:44:12.326 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pdu" 2020-03-11T22:44:12.326 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.326 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.326 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 6, 2020-03-11T22:44:12.326 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pod" 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 7, 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "room" 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 8, 2020-03-11T22:44:12.327 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "datacenter" 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 9, 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "zone" 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 10, 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "region" 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "root" 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: "buckets": [ 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -1, 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default", 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:12.330 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -2, 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default~hdd", 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:44:12.332 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:12.333 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130", 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:44:12.334 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:44:12.335 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:12.335 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:12.335 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:12.335 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:12.335 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.335 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:12.335 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.336 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.337 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:44:12.338 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130~hdd", 2020-03-11T22:44:12.338 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:44:12.338 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:44:12.338 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:12.338 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:12.338 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:12.338 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:12.339 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.339 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:12.339 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:12.339 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:12.339 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.339 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.339 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:12.340 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:12.340 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:44:12.340 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.340 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.340 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:12.340 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:12.340 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:44:12.341 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.341 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.341 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.341 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.341 INFO:tasks.workunit.client.0.smithi130.stdout: "rules": [ 2020-03-11T22:44:12.341 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.341 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 0, 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "replicated_rule", 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 0, 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 1, 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 10, 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.342 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_firstn", 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:44:12.343 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.344 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 1, 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "test", 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 1, 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 3, 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 3, 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 3, 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.345 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 5 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "set_choose_tries", 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 100 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.346 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:44:12.347 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:44:12.347 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:44:12.347 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.347 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.347 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_indep", 2020-03-11T22:44:12.347 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:44:12.347 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.348 INFO:tasks.workunit.client.0.smithi130.stdout: "tunables": { 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_tries": 0, 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_total_tries": 50, 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_stable": 1, 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "straw_calc_version": 1, 2020-03-11T22:44:12.349 INFO:tasks.workunit.client.0.smithi130.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:44:12.350 INFO:tasks.workunit.client.0.smithi130.stdout: "profile": "jewel", 2020-03-11T22:44:12.350 INFO:tasks.workunit.client.0.smithi130.stdout: "optimal_tunables": 1, 2020-03-11T22:44:12.350 INFO:tasks.workunit.client.0.smithi130.stdout: "legacy_tunables": 0, 2020-03-11T22:44:12.350 INFO:tasks.workunit.client.0.smithi130.stdout: "minimum_required_version": "jewel", 2020-03-11T22:44:12.350 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables": 1, 2020-03-11T22:44:12.350 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables2": 1, 2020-03-11T22:44:12.350 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v2_rules": 1, 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables3": 1, 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v3_rules": 0, 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v4_buckets": 1, 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables5": 1, 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v5_rules": 0 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_args": {} 2020-03-11T22:44:12.351 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.352 INFO:tasks.workunit.client.0.smithi130.stdout: "fsmap": { 2020-03-11T22:44:12.352 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:44:12.352 INFO:tasks.workunit.client.0.smithi130.stdout: "default_fscid": -1, 2020-03-11T22:44:12.352 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": { 2020-03-11T22:44:12.352 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": {}, 2020-03-11T22:44:12.352 INFO:tasks.workunit.client.0.smithi130.stdout: "ro_compat": {}, 2020-03-11T22:44:12.352 INFO:tasks.workunit.client.0.smithi130.stdout: "incompat": { 2020-03-11T22:44:12.353 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_1": "base v0.20", 2020-03-11T22:44:12.353 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:44:12.353 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:44:12.353 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:44:12.353 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:44:12.353 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:44:12.353 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_8": "no anchor table", 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_9": "file layout v2", 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_flags": { 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: "enable_multiple": false, 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: "ever_enabled_multiple": false 2020-03-11T22:44:12.354 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: "standbys": [], 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: "filesystems": [] 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: "auth": { 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:44:12.355 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 5, 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: "num_secrets": 3 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_active": 1, 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_unknown": 0, 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osd": 3, 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_sum": { 2020-03-11T22:44:12.356 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:44:12.357 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48430, 2020-03-11T22:44:12.357 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2020-03-11T22:44:12.357 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 1, 2020-03-11T22:44:12.357 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 303, 2020-03-11T22:44:12.357 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:12.357 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:44:12.357 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:44:12.358 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:44:12.359 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:12.359 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:12.359 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:12.359 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:12.359 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:12.359 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:12.359 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:44:12.360 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:12.361 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:12.361 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:12.361 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:12.361 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:12.361 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:12.361 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:12.361 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:44:12.362 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:44:12.363 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 0, 2020-03-11T22:44:12.363 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 0, 2020-03-11T22:44:12.363 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:12.363 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:12.363 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:12.363 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:44:12.363 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 18, 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 3, 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 3, 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 0 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.364 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum": { 2020-03-11T22:44:12.365 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:44:12.365 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:44:12.365 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:44:12.365 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:44:12.365 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:12.365 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:12.365 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3179584, 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 33600, 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 10, 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145717, 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393216, 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:44:12.366 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318866653184, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 34406400, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 1525967, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 10752, 2020-03-11T22:44:12.367 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221214720 2020-03-11T22:44:12.368 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.368 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:44:12.368 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:12.368 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:12.368 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:12.368 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:44:12.368 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:44:12.369 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.370 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:44:12.370 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:44:12.370 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.370 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum_by_class": { 2020-03-11T22:44:12.370 INFO:tasks.workunit.client.0.smithi130.stdout: "hdd": { 2020-03-11T22:44:12.370 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:44:12.370 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:44:12.371 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:44:12.371 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:44:12.371 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:12.371 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:12.371 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:44:12.371 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3179584, 2020-03-11T22:44:12.371 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 33600, 2020-03-11T22:44:12.372 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 10, 2020-03-11T22:44:12.372 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145717, 2020-03-11T22:44:12.372 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311393216, 2020-03-11T22:44:12.372 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:44:12.372 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:44:12.372 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318866653184, 2020-03-11T22:44:12.372 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:12.373 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 34406400, 2020-03-11T22:44:12.373 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 1525967, 2020-03-11T22:44:12.373 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:12.373 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:12.373 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:12.373 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 10752, 2020-03-11T22:44:12.373 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221214720 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:44:12.374 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.375 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_stats": [ 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: "poolid": 1, 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:44:12.376 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:44:12.377 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48430, 2020-03-11T22:44:12.377 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2020-03-11T22:44:12.377 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 1, 2020-03-11T22:44:12.377 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 303, 2020-03-11T22:44:12.377 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:12.377 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:44:12.377 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:12.378 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:12.378 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:12.378 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:12.378 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:44:12.378 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:44:12.378 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:44:12.379 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:44:12.379 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:44:12.379 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:12.379 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:12.379 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:12.379 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:12.379 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:12.380 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:12.380 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:44:12.380 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:12.380 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:12.380 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:44:12.380 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:12.381 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:12.382 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:12.382 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:12.382 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:12.382 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:12.382 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:44:12.382 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:44:12.383 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.383 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:44:12.383 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:44:12.383 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:44:12.383 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:44:12.383 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 19857408, 2020-03-11T22:44:12.383 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 1241088, 2020-03-11T22:44:12.384 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:12.384 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:12.384 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:12.384 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:44:12.384 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:44:12.384 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.384 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 18, 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 3, 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 3, 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 3 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_stats": [ 2020-03-11T22:44:12.385 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.386 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:12.386 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 146028888077 2020-03-11T22:44:12.386 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.386 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.386 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:12.386 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 38654705682 2020-03-11T22:44:12.386 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 150323855373 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_state": [ 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.387 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 1 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_osd": [ 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:44:12.388 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:44:12.389 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:12.389 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.389 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.389 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2147483647, 2020-03-11T22:44:12.389 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:44:12.389 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:44:12.389 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 0, 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.390 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.391 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:12.391 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 1, 2020-03-11T22:44:12.391 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:44:12.391 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:12.391 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.391 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.391 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [ 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": {} 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: "servicemap": { 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:44:12.392 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:42:13.715486+0000", 2020-03-11T22:44:12.393 INFO:tasks.workunit.client.0.smithi130.stdout: "services": {} 2020-03-11T22:44:12.393 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:12.393 INFO:tasks.workunit.client.0.smithi130.stdout: "paxos": { 2020-03-11T22:44:12.393 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:44:12.393 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 339, 2020-03-11T22:44:12.393 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pn": 0, 2020-03-11T22:44:12.393 INFO:tasks.workunit.client.0.smithi130.stdout: "accepted_pn": 0 2020-03-11T22:44:12.394 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:12.394 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:44:12.395 INFO:tasks.workunit.client.0.smithi130.stderr:report 33095052 2020-03-11T22:44:12.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:44:12.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:12.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:44:12.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:12.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:12.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:12.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:12.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:12.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:12.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:44:12.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:12.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:12.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:44:12.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:12.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:44:12.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:44:12.850 INFO:tasks.workunit.client.0.smithi130.stderr:PG '1.0' not found 2020-03-11T22:44:13.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:44:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:44:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:44:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:44:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:44:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:44:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:44:13.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:44:13.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:44:13.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:13.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:13.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:13.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:13.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:13.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:13.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:13.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:13.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:44:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:44:13.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:44:13.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:13.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:44:13.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:44:13.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:13.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:13.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:13.399 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:13.397+0000 7f5768d5bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:13.401 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:13.399+0000 7f5768d5bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:13.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:13.401+0000 7f5768d5bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:13.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:44:13.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:13.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:13.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:13.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:13.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:13.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:13.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:13.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:13.727 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:44:13.920 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:13.917+0000 7f5768d5bec0 -1 Falling back to public interface 2020-03-11T22:44:14.078 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 34 up_thru 39 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/48667,v1:127.0.0.1:6803/48667] [v2:127.0.0.1:6804/48667,v1:127.0.0.1:6805/48667] exists,up d34a39d8-760e-4854-8c86-11e878da47ec 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:14.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:14.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:14.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:14.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:14.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:44:14.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:14.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:14.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:44:14.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:14.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:44:14.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:44:14.824 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:14.822+0000 7f5768d5bec0 -1 osd.0 40 log_to_monitors {default=true} 2020-03-11T22:44:14.932 INFO:tasks.workunit.client.0.smithi130.stderr:PG '1.0' not found 2020-03-11T22:44:15.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:44:15.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:44:15.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:15.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:15.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:44:15.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:15.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:44:15.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:15.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:15.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:15.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:15.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:44:15.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:44:15.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:15.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:15.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:15.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:44:15.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:15.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:44:15.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:44:15.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:15.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:44:15.456 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:44:15.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:44:15.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:15.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:15.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:15.476+0000 7fb136824ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:15.482 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:15.480+0000 7fb136824ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:15.484 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:15.481+0000 7fb136824ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:15.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:44:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:15.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:15.805 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:44:16.000 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:15.998+0000 7fb136824ec0 -1 Falling back to public interface 2020-03-11T22:44:16.184 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [13,27) [v2:127.0.0.1:6810/48861,v1:127.0.0.1:6811/48861] [v2:127.0.0.1:6812/48861,v1:127.0.0.1:6813/48861] exists,up 9bd7863a-9ae2-4649-8393-d7b70b4864ee 2020-03-11T22:44:16.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:16.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:16.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:16.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:44:16.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:16.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:16.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:44:16.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:16.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:44:16.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:44:16.883 INFO:tasks.workunit.client.0.smithi130.stderr:PG '1.0' not found 2020-03-11T22:44:16.958 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:16.956+0000 7fb136824ec0 -1 osd.2 40 log_to_monitors {default=true} 2020-03-11T22:44:17.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:44:17.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:17.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:17.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:17.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:17.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:17.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:17.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:17.402 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:17.402 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:17.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:17.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:17.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:17.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:17.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:17.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:17.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:17.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:17.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:17.862 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:44:17.862 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2020-03-11T22:44:17.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:17.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:17.862 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:17.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=180388626435 2020-03-11T22:44:17.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435' 2020-03-11T22:44:17.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:17.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:18.086 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:44:18.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:44:18.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435 1-' 2020-03-11T22:44:18.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:18.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:18.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=193273528323 2020-03-11T22:44:18.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-180388626435 1- 2-193273528323' 2020-03-11T22:44:18.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:18.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-180388626435 2020-03-11T22:44:18.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:18.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-180388626435 2020-03-11T22:44:18.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:18.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=180388626435 2020-03-11T22:44:18.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 180388626435' 2020-03-11T22:44:18.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:18.228 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 180388626435 2020-03-11T22:44:18.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 180388626435 2020-03-11T22:44:18.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:19.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:19.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:19.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 180388626435 2020-03-11T22:44:19.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:20.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:44:20.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:21.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 180388626435 -lt 180388626435 2020-03-11T22:44:21.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:21.278 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:21.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:44:21.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:21.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:44:21.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:21.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:44:21.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:44:21.281 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 2020-03-11T22:44:21.281 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:21.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705682 -lt 2020-03-11T22:44:21.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705682: unary operator expected 2020-03-11T22:44:21.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:21.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-193273528323 2020-03-11T22:44:21.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:21.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:21.627 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-193273528323 2020-03-11T22:44:21.627 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:21.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=193273528323 2020-03-11T22:44:21.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2020-03-11T22:44:21.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:21.628 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 193273528323 2020-03-11T22:44:21.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 193273528323 -lt 193273528323 2020-03-11T22:44:21.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:21.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:21.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:22.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:22.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:22.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:22.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:22.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:22.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:22.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:22.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:22.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:22.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:22.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:22.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:44:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:44:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:44:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:44:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:23.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:23.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:23.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:23.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:23.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:23.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:23.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:23.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:23.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:23.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:24.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:24.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:24.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:24.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:24.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:24.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:24.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:24.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:24.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:24.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:24.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:24.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:24.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:24.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:24.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:24.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:25.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:25.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:25.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:25.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:25.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:25.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:25.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:25.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:25.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:25.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:25.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:25.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:25.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:25.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:25.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:25.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:25.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:25.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:26.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:26.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:26.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:26.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:26.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:26.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:26.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:26.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:26.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:26.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:26.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:26.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:27.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:27.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:27.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:27.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:27.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:27.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:27.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:27.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:27.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:27.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:27.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:27.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:27.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:27.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:28.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:28.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:28.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:28.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:28.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:28.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:28.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:28.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:28.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:28.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:28.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:28.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:29.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:29.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:29.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:29.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:29.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:29.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:29.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:29.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:29.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:29.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:29.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:29.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:44:29.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:29.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:44:31.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:31.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:31.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:31.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:31.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:31.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:31.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:31.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:31.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:31.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:31.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:31.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:31.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:31.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:31.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:31.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:31.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:31.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:31.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:31.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:31.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:32.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:32.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:32.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:44:32.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:32.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:44:35.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:35.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:35.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:35.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:35.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:35.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:35.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:35.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:35.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:35.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:35.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:35.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:36.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:36.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:36.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:36.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:36.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:36.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:36.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:36.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:36.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:36.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:36.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:36.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:44:36.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:36.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:44:43.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:43.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:43.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:43.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:43.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:43.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:43.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:43.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:43.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:43.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:43.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:44.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:44.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:44.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:44.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:44.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:44.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:44.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:44.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:44.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:44:44.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:44.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:44:51.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:51.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:51.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:51.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:51.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:51.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:51.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:51.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:52.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:52.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:52.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:52.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:52.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:52.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:52.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:52.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:52.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:52.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:52.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:52.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:52.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:53.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:53.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:53.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:44:53.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:44:53.455 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:44:53.455 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_fingerprint": "63286032-48e7-4f61-99d7-1878700996a8", 2020-03-11T22:44:53.455 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:44:53.455 INFO:tasks.workunit.client.0.smithi130.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:44:53.455 INFO:tasks.workunit.client.0.smithi130.stdout: "timestamp": "2020-03-11T22:44:53.451608+0000", 2020-03-11T22:44:53.455 INFO:tasks.workunit.client.0.smithi130.stdout: "tag": "", 2020-03-11T22:44:53.456 INFO:tasks.workunit.client.0.smithi130.stdout: "health": { 2020-03-11T22:44:53.456 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "HEALTH_WARN", 2020-03-11T22:44:53.456 INFO:tasks.workunit.client.0.smithi130.stdout: "checks": { 2020-03-11T22:44:53.456 INFO:tasks.workunit.client.0.smithi130.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:44:53.456 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:53.456 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:53.456 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:44:53.457 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 17 2020-03-11T22:44:53.457 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.457 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [], 2020-03-11T22:44:53.457 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:53.457 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.457 INFO:tasks.workunit.client.0.smithi130.stdout: "OSD_DOWN": { 2020-03-11T22:44:53.457 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 osds down", 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "osd.1 (root=default,host=smithi130) is down" 2020-03-11T22:44:53.458 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.459 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.459 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:53.459 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.459 INFO:tasks.workunit.client.0.smithi130.stdout: "PG_DEGRADED": { 2020-03-11T22:44:53.459 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:53.459 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:53.459 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.460 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:53.461 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.461 INFO:tasks.workunit.client.0.smithi130.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:44:53.461 INFO:tasks.workunit.client.0.smithi130.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:53.461 INFO:tasks.workunit.client.0.smithi130.stdout: "summary": { 2020-03-11T22:44:53.461 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:44:53.461 INFO:tasks.workunit.client.0.smithi130.stdout: "count": 1 2020-03-11T22:44:53.461 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: "detail": [ 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.462 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.463 INFO:tasks.workunit.client.0.smithi130.stdout: "muted": false 2020-03-11T22:44:53.463 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.463 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.463 INFO:tasks.workunit.client.0.smithi130.stdout: "mutes": [] 2020-03-11T22:44:53.463 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.463 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_first_committed": 1, 2020-03-11T22:44:53.463 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap_last_committed": 1, 2020-03-11T22:44:53.464 INFO:tasks.workunit.client.0.smithi130.stdout: "monmap": { 2020-03-11T22:44:53.464 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:44:53.464 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:44:53.464 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:42:13.015911Z", 2020-03-11T22:44:53.464 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:42:13.015911Z", 2020-03-11T22:44:53.464 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release": 15, 2020-03-11T22:44:53.464 INFO:tasks.workunit.client.0.smithi130.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:44:53.465 INFO:tasks.workunit.client.0.smithi130.stdout: "features": { 2020-03-11T22:44:53.465 INFO:tasks.workunit.client.0.smithi130.stdout: "persistent": [ 2020-03-11T22:44:53.465 INFO:tasks.workunit.client.0.smithi130.stdout: "kraken", 2020-03-11T22:44:53.465 INFO:tasks.workunit.client.0.smithi130.stdout: "luminous", 2020-03-11T22:44:53.465 INFO:tasks.workunit.client.0.smithi130.stdout: "mimic", 2020-03-11T22:44:53.465 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap-prune", 2020-03-11T22:44:53.465 INFO:tasks.workunit.client.0.smithi130.stdout: "nautilus", 2020-03-11T22:44:53.466 INFO:tasks.workunit.client.0.smithi130.stdout: "octopus" 2020-03-11T22:44:53.466 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.466 INFO:tasks.workunit.client.0.smithi130.stdout: "optional": [] 2020-03-11T22:44:53.466 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.466 INFO:tasks.workunit.client.0.smithi130.stdout: "mons": [ 2020-03-11T22:44:53.466 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.466 INFO:tasks.workunit.client.0.smithi130.stdout: "rank": 0, 2020-03-11T22:44:53.467 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "a", 2020-03-11T22:44:53.467 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:53.467 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.467 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.467 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.467 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:44:53.467 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 0 2020-03-11T22:44:53.468 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.468 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.468 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.468 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:44:53.468 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:44:53.468 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 0, 2020-03-11T22:44:53.469 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 0 2020-03-11T22:44:53.469 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.469 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.469 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.469 INFO:tasks.workunit.client.0.smithi130.stdout: "quorum": [ 2020-03-11T22:44:53.469 INFO:tasks.workunit.client.0.smithi130.stdout: 0 2020-03-11T22:44:53.469 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap": { 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 46, 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "fsid": "70af20e8-df52-4c90-ac53-59b9c01e7b62", 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "created": "2020-03-11T22:42:13.715168+0000", 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:44:17.434061+0000", 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "last_up_change": "2020-03-11T22:44:17.225126+0000", 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "last_in_change": "2020-03-11T22:43:21.454158+0000", 2020-03-11T22:44:53.470 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:44:53.471 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_num": 5800384, 2020-03-11T22:44:53.471 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_set": [ 2020-03-11T22:44:53.471 INFO:tasks.workunit.client.0.smithi130.stdout: "nodown", 2020-03-11T22:44:53.471 INFO:tasks.workunit.client.0.smithi130.stdout: "noin", 2020-03-11T22:44:53.471 INFO:tasks.workunit.client.0.smithi130.stdout: "noout", 2020-03-11T22:44:53.471 INFO:tasks.workunit.client.0.smithi130.stdout: "pglog_hardlimit", 2020-03-11T22:44:53.472 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snapdirs", 2020-03-11T22:44:53.472 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_deletes", 2020-03-11T22:44:53.472 INFO:tasks.workunit.client.0.smithi130.stdout: "sortbitwise" 2020-03-11T22:44:53.472 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.472 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_version": 8, 2020-03-11T22:44:53.472 INFO:tasks.workunit.client.0.smithi130.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:44:53.472 INFO:tasks.workunit.client.0.smithi130.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:44:53.473 INFO:tasks.workunit.client.0.smithi130.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:44:53.473 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_snapshot": "", 2020-03-11T22:44:53.473 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_max": 1, 2020-03-11T22:44:53.473 INFO:tasks.workunit.client.0.smithi130.stdout: "max_osd": 3, 2020-03-11T22:44:53.473 INFO:tasks.workunit.client.0.smithi130.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:44:53.473 INFO:tasks.workunit.client.0.smithi130.stdout: "min_compat_client": "jewel", 2020-03-11T22:44:53.473 INFO:tasks.workunit.client.0.smithi130.stdout: "require_osd_release": "octopus", 2020-03-11T22:44:53.474 INFO:tasks.workunit.client.0.smithi130.stdout: "pools": [ 2020-03-11T22:44:53.474 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.474 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:44:53.474 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_name": "test", 2020-03-11T22:44:53.474 INFO:tasks.workunit.client.0.smithi130.stdout: "create_time": "2020-03-11T22:42:35.958624+0000", 2020-03-11T22:44:53.474 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": 16389, 2020-03-11T22:44:53.474 INFO:tasks.workunit.client.0.smithi130.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 3, 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "size": 3, 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 2, 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_rule": 1, 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "object_hash": 2, 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num": 1, 2020-03-11T22:44:53.475 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num": 1, 2020-03-11T22:44:53.476 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_placement_num_target": 1, 2020-03-11T22:44:53.476 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_target": 1, 2020-03-11T22:44:53.476 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_num_pending": 1, 2020-03-11T22:44:53.476 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pg_merge_meta": { 2020-03-11T22:44:53.476 INFO:tasks.workunit.client.0.smithi130.stdout: "source_pgid": "0.0", 2020-03-11T22:44:53.476 INFO:tasks.workunit.client.0.smithi130.stdout: "ready_epoch": 0, 2020-03-11T22:44:53.476 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 0, 2020-03-11T22:44:53.477 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 0, 2020-03-11T22:44:53.477 INFO:tasks.workunit.client.0.smithi130.stdout: "source_version": "0'0", 2020-03-11T22:44:53.477 INFO:tasks.workunit.client.0.smithi130.stdout: "target_version": "0'0" 2020-03-11T22:44:53.477 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.477 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "28", 2020-03-11T22:44:53.477 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend": "0", 2020-03-11T22:44:53.477 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:44:53.478 INFO:tasks.workunit.client.0.smithi130.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:44:53.478 INFO:tasks.workunit.client.0.smithi130.stdout: "auid": 0, 2020-03-11T22:44:53.478 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_mode": "pool", 2020-03-11T22:44:53.478 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_seq": 1, 2020-03-11T22:44:53.478 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_epoch": 28, 2020-03-11T22:44:53.478 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_snaps": [ 2020-03-11T22:44:53.478 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.479 INFO:tasks.workunit.client.0.smithi130.stdout: "snapid": 1, 2020-03-11T22:44:53.479 INFO:tasks.workunit.client.0.smithi130.stdout: "stamp": "2020-03-11T22:43:03.056898+0000", 2020-03-11T22:44:53.479 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "snap1" 2020-03-11T22:44:53.479 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.479 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.479 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps": "[]", 2020-03-11T22:44:53.479 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_bytes": 0, 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "quota_max_objects": 0, 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "tiers": [], 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "tier_of": -1, 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "read_tier": -1, 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "write_tier": -1, 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_mode": "none", 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_bytes": 0, 2020-03-11T22:44:53.480 INFO:tasks.workunit.client.0.smithi130.stdout: "target_max_objects": 0, 2020-03-11T22:44:53.481 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:44:53.481 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:44:53.481 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:44:53.481 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_flush_age": 0, 2020-03-11T22:44:53.481 INFO:tasks.workunit.client.0.smithi130.stdout: "cache_min_evict_age": 0, 2020-03-11T22:44:53.481 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:44:53.481 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_params": { 2020-03-11T22:44:53.482 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "none" 2020-03-11T22:44:53.482 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.482 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_period": 0, 2020-03-11T22:44:53.482 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_count": 0, 2020-03-11T22:44:53.482 INFO:tasks.workunit.client.0.smithi130.stdout: "use_gmt_hitset": true, 2020-03-11T22:44:53.482 INFO:tasks.workunit.client.0.smithi130.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:44:53.482 INFO:tasks.workunit.client.0.smithi130.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "grade_table": [], 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "stripe_width": 8192, 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "expected_num_objects": 0, 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "fast_read": false, 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "options": {}, 2020-03-11T22:44:53.483 INFO:tasks.workunit.client.0.smithi130.stdout: "application_metadata": {} 2020-03-11T22:44:53.484 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.484 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.484 INFO:tasks.workunit.client.0.smithi130.stdout: "osds": [ 2020-03-11T22:44:53.484 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.484 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:53.484 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "d34a39d8-760e-4854-8c86-11e878da47ec", 2020-03-11T22:44:53.484 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:44:53.485 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:44:53.485 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:44:53.485 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:44:53.485 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 34, 2020-03-11T22:44:53.486 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 40, 2020-03-11T22:44:53.486 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 42, 2020-03-11T22:44:53.486 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 45, 2020-03-11T22:44:53.486 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 41, 2020-03-11T22:44:53.486 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.487 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.488 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.488 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:44:53.488 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.488 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.488 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.488 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.488 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:44:53.489 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.489 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.489 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.489 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:44:53.489 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.489 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.489 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.490 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.490 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:44:53.490 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.490 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.490 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.490 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.490 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:53.491 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.491 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.491 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.491 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:44:53.491 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.491 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.491 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:53.492 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.493 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.493 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.493 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:44:53.493 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.493 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.493 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.493 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50310 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6803/50310", 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6805/50310", 2020-03-11T22:44:53.494 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50310", 2020-03-11T22:44:53.495 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50310", 2020-03-11T22:44:53.495 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:44:53.495 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:44:53.495 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:44:53.495 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.495 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.495 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.496 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:53.496 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5", 2020-03-11T22:44:53.496 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 0, 2020-03-11T22:44:53.496 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:44:53.496 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:44:53.496 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:44:53.496 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 0, 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 0, 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 9, 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 23, 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 30, 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.497 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.498 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.498 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:53.498 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.498 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.498 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.498 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.498 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.499 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.500 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:53.500 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.500 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.500 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.500 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.500 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:53.500 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.501 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.501 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.501 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.501 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:53.501 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.501 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.501 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.502 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.503 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.503 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.503 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:53.503 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.503 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.503 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.503 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 43909 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.504 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.505 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.505 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/43909", 2020-03-11T22:44:53.505 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/43909", 2020-03-11T22:44:53.505 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43909", 2020-03-11T22:44:53.505 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43909", 2020-03-11T22:44:53.505 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:44:53.505 INFO:tasks.workunit.client.0.smithi130.stdout: "exists" 2020-03-11T22:44:53.506 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.506 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.506 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.506 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:53.506 INFO:tasks.workunit.client.0.smithi130.stdout: "uuid": "9bd7863a-9ae2-4649-8393-d7b70b4864ee", 2020-03-11T22:44:53.506 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 1, 2020-03-11T22:44:53.506 INFO:tasks.workunit.client.0.smithi130.stdout: "in": 1, 2020-03-11T22:44:53.507 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 1, 2020-03-11T22:44:53.507 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_affinity": 1, 2020-03-11T22:44:53.507 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_begin": 35, 2020-03-11T22:44:53.507 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_end": 40, 2020-03-11T22:44:53.507 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 45, 2020-03-11T22:44:53.507 INFO:tasks.workunit.client.0.smithi130.stdout: "up_thru": 0, 2020-03-11T22:44:53.508 INFO:tasks.workunit.client.0.smithi130.stdout: "down_at": 44, 2020-03-11T22:44:53.508 INFO:tasks.workunit.client.0.smithi130.stdout: "lost_at": 0, 2020-03-11T22:44:53.508 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addrs": { 2020-03-11T22:44:53.508 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.508 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.508 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.508 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.509 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.510 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.510 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addrs": { 2020-03-11T22:44:53.510 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.510 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.510 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.510 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:53.510 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.511 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.512 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:53.512 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.512 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.512 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.512 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:53.512 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.512 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.513 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.513 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.513 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:53.513 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.513 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.513 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.513 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: "addrvec": [ 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v2", 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.514 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "v1", 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi130.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi130.stdout: "nonce": 50542 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.515 INFO:tasks.workunit.client.0.smithi130.stdout: "public_addr": "127.0.0.1:6811/50542", 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: "cluster_addr": "127.0.0.1:6813/50542", 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_back_addr": "127.0.0.1:6817/50542", 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: "heartbeat_front_addr": "127.0.0.1:6815/50542", 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: "state": [ 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: "exists", 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: "up" 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.516 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.517 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.517 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_xinfo": [ 2020-03-11T22:44:53.517 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.517 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:53.517 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:44:14.913694+0000", 2020-03-11T22:44:53.517 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:44:53.517 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:20.806050+0000", 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:53.518 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:43:14.428449+0000", 2020-03-11T22:44:53.519 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:44:53.519 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:44:53.519 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:44:53.519 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:44:53.519 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:27.028030+0000", 2020-03-11T22:44:53.519 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:44:53.519 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.520 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.520 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:53.520 INFO:tasks.workunit.client.0.smithi130.stdout: "down_stamp": "2020-03-11T22:44:17.031503+0000", 2020-03-11T22:44:53.520 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_probability": 0, 2020-03-11T22:44:53.520 INFO:tasks.workunit.client.0.smithi130.stdout: "laggy_interval": 0, 2020-03-11T22:44:53.520 INFO:tasks.workunit.client.0.smithi130.stdout: "features": 4540138292836696063, 2020-03-11T22:44:53.520 INFO:tasks.workunit.client.0.smithi130.stdout: "old_weight": 0, 2020-03-11T22:44:53.521 INFO:tasks.workunit.client.0.smithi130.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:33.196975+0000", 2020-03-11T22:44:53.521 INFO:tasks.workunit.client.0.smithi130.stdout: "dead_epoch": 0 2020-03-11T22:44:53.521 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.521 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.521 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap": [], 2020-03-11T22:44:53.521 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_upmap_items": [], 2020-03-11T22:44:53.521 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_temp": [], 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "primary_temp": [], 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "blacklist": {}, 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "erasure_code_profiles": { 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "default": { 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "2", 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:44:53.522 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van" 2020-03-11T22:44:53.523 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.523 INFO:tasks.workunit.client.0.smithi130.stdout: "myprofile": { 2020-03-11T22:44:53.523 INFO:tasks.workunit.client.0.smithi130.stdout: "crush-device-class": "", 2020-03-11T22:44:53.523 INFO:tasks.workunit.client.0.smithi130.stdout: "crush-failure-domain": "osd", 2020-03-11T22:44:53.523 INFO:tasks.workunit.client.0.smithi130.stdout: "crush-root": "default", 2020-03-11T22:44:53.523 INFO:tasks.workunit.client.0.smithi130.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:44:53.523 INFO:tasks.workunit.client.0.smithi130.stdout: "k": "2", 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: "m": "1", 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: "plugin": "jerasure", 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: "technique": "reed_sol_van", 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: "w": "8" 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: "removed_snaps_queue": [], 2020-03-11T22:44:53.524 INFO:tasks.workunit.client.0.smithi130.stdout: "new_removed_snaps": [], 2020-03-11T22:44:53.525 INFO:tasks.workunit.client.0.smithi130.stdout: "new_purged_snaps": [], 2020-03-11T22:44:53.525 INFO:tasks.workunit.client.0.smithi130.stdout: "crush_node_flags": {}, 2020-03-11T22:44:53.525 INFO:tasks.workunit.client.0.smithi130.stdout: "device_class_flags": {} 2020-03-11T22:44:53.525 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.525 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_metadata": [ 2020-03-11T22:44:53.525 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.525 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:53.526 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:44:53.526 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6804/50310,v1:127.0.0.1:6805/50310]", 2020-03-11T22:44:53.526 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:44:53.526 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:44:53.526 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:53.526 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:53.526 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:53.527 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:53.528 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:53.528 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:44:53.528 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:53.528 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:53.528 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:53.528 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:44:53.528 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:44:53.529 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:53.529 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:44:53.529 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:44:53.529 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:53.529 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:44:53.529 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6802/50310,v1:127.0.0.1:6803/50310]", 2020-03-11T22:44:53.529 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50310,v1:127.0.0.1:6809/50310]", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50310,v1:127.0.0.1:6807/50310]", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:53.530 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:44:53.531 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:53.531 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:53.531 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:44:53.531 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:44:53.531 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:53.531 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:44:53.531 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.532 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.532 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:53.532 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:44:53.532 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/43909,v1:127.0.0.1:6813/43909]", 2020-03-11T22:44:53.532 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:44:53.532 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:44:53.532 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:53.533 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:53.533 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:53.533 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:53.533 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:53.533 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:53.533 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:53.533 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:53.534 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:53.535 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:44:53.535 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:44:53.535 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:53.535 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:44:53.535 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:44:53.535 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:53.536 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:44:53.536 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/43909,v1:127.0.0.1:6811/43909]", 2020-03-11T22:44:53.536 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:44:53.536 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43909,v1:127.0.0.1:6817/43909]", 2020-03-11T22:44:53.536 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43909,v1:127.0.0.1:6815/43909]", 2020-03-11T22:44:53.536 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:44:53.536 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:44:53.537 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:53.537 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:53.537 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:53.537 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:44:53.537 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:53.537 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:53.537 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: "arch": "x86_64", 2020-03-11T22:44:53.538 INFO:tasks.workunit.client.0.smithi130.stdout: "back_addr": "[v2:127.0.0.1:6812/50542,v1:127.0.0.1:6813/50542]", 2020-03-11T22:44:53.539 INFO:tasks.workunit.client.0.smithi130.stdout: "back_iface": "", 2020-03-11T22:44:53.539 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs": "1", 2020-03-11T22:44:53.539 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:53.539 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:53.539 INFO:tasks.workunit.client.0.smithi130.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:53.539 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:53.539 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:53.540 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:53.540 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:53.540 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:44:53.540 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:53.540 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:53.540 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:53.540 INFO:tasks.workunit.client.0.smithi130.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:53.541 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_release": "octopus", 2020-03-11T22:44:53.541 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:53.541 INFO:tasks.workunit.client.0.smithi130.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:53.541 INFO:tasks.workunit.client.0.smithi130.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:53.541 INFO:tasks.workunit.client.0.smithi130.stdout: "default_device_class": "hdd", 2020-03-11T22:44:53.541 INFO:tasks.workunit.client.0.smithi130.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XK59", 2020-03-11T22:44:53.541 INFO:tasks.workunit.client.0.smithi130.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:53.542 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": "sda", 2020-03-11T22:44:53.542 INFO:tasks.workunit.client.0.smithi130.stdout: "distro": "centos", 2020-03-11T22:44:53.542 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:53.542 INFO:tasks.workunit.client.0.smithi130.stdout: "distro_version": "8", 2020-03-11T22:44:53.542 INFO:tasks.workunit.client.0.smithi130.stdout: "front_addr": "[v2:127.0.0.1:6810/50542,v1:127.0.0.1:6811/50542]", 2020-03-11T22:44:53.542 INFO:tasks.workunit.client.0.smithi130.stdout: "front_iface": "", 2020-03-11T22:44:53.542 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/50542,v1:127.0.0.1:6817/50542]", 2020-03-11T22:44:53.543 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/50542,v1:127.0.0.1:6815/50542]", 2020-03-11T22:44:53.543 INFO:tasks.workunit.client.0.smithi130.stdout: "hostname": "smithi130", 2020-03-11T22:44:53.543 INFO:tasks.workunit.client.0.smithi130.stdout: "journal_rotational": "1", 2020-03-11T22:44:53.543 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:53.543 INFO:tasks.workunit.client.0.smithi130.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:53.543 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:53.544 INFO:tasks.workunit.client.0.smithi130.stdout: "mem_total_kb": "32655816", 2020-03-11T22:44:53.544 INFO:tasks.workunit.client.0.smithi130.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:53.544 INFO:tasks.workunit.client.0.smithi130.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:53.544 INFO:tasks.workunit.client.0.smithi130.stdout: "os": "Linux", 2020-03-11T22:44:53.545 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:44:53.545 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:53.545 INFO:tasks.workunit.client.0.smithi130.stdout: "rotational": "1" 2020-03-11T22:44:53.545 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.545 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.546 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_first_committed": 1, 2020-03-11T22:44:53.546 INFO:tasks.workunit.client.0.smithi130.stdout: "osdmap_last_committed": 46, 2020-03-11T22:44:53.546 INFO:tasks.workunit.client.0.smithi130.stdout: "crushmap": { 2020-03-11T22:44:53.546 INFO:tasks.workunit.client.0.smithi130.stdout: "devices": [ 2020-03-11T22:44:53.546 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.546 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:53.546 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.0", 2020-03-11T22:44:53.547 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:44:53.547 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.547 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.547 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:53.547 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.1", 2020-03-11T22:44:53.547 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:44:53.547 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.548 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.548 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:53.548 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd.2", 2020-03-11T22:44:53.548 INFO:tasks.workunit.client.0.smithi130.stdout: "class": "hdd" 2020-03-11T22:44:53.548 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.548 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.548 INFO:tasks.workunit.client.0.smithi130.stdout: "types": [ 2020-03-11T22:44:53.549 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.549 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 0, 2020-03-11T22:44:53.549 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "osd" 2020-03-11T22:44:53.549 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.549 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.549 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "host" 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 2, 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "chassis" 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.550 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 3, 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "rack" 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 4, 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "row" 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.551 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 5, 2020-03-11T22:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pdu" 2020-03-11T22:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 6, 2020-03-11T22:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "pod" 2020-03-11T22:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 7, 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "room" 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 8, 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "datacenter" 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.554 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 9, 2020-03-11T22:44:53.554 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "zone" 2020-03-11T22:44:53.554 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.554 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.554 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 10, 2020-03-11T22:44:53.554 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "region" 2020-03-11T22:44:53.554 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.555 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.555 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:44:53.555 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "root" 2020-03-11T22:44:53.555 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.556 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.556 INFO:tasks.workunit.client.0.smithi130.stdout: "buckets": [ 2020-03-11T22:44:53.556 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.556 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -1, 2020-03-11T22:44:53.556 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default", 2020-03-11T22:44:53.556 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:44:53.557 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:44:53.557 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:53.557 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:53.557 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:53.557 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:53.557 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.557 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:44:53.558 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:53.558 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:53.558 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.558 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.558 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.558 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.558 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -2, 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "default~hdd", 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 11, 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "root", 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:53.559 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.560 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -3, 2020-03-11T22:44:53.561 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130", 2020-03-11T22:44:53.561 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:44:53.561 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:44:53.561 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:53.561 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:53.561 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:53.561 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:53.562 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.563 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.564 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.564 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.564 INFO:tasks.workunit.client.0.smithi130.stdout: "id": -4, 2020-03-11T22:44:53.564 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "smithi130~hdd", 2020-03-11T22:44:53.564 INFO:tasks.workunit.client.0.smithi130.stdout: "type_id": 1, 2020-03-11T22:44:53.564 INFO:tasks.workunit.client.0.smithi130.stdout: "type_name": "host", 2020-03-11T22:44:53.564 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 19206, 2020-03-11T22:44:53.565 INFO:tasks.workunit.client.0.smithi130.stdout: "alg": "straw2", 2020-03-11T22:44:53.565 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": "rjenkins1", 2020-03-11T22:44:53.565 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:44:53.565 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.565 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 0, 2020-03-11T22:44:53.565 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:53.566 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 0 2020-03-11T22:44:53.566 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.566 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.566 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 1, 2020-03-11T22:44:53.566 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:53.566 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 1 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: "id": 2, 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: "weight": 6402, 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: "pos": 2 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.567 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.568 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.568 INFO:tasks.workunit.client.0.smithi130.stdout: "rules": [ 2020-03-11T22:44:53.568 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.568 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 0, 2020-03-11T22:44:53.568 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "replicated_rule", 2020-03-11T22:44:53.568 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 0, 2020-03-11T22:44:53.568 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 1, 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 1, 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 10, 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:44:53.569 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.570 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.570 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_firstn", 2020-03-11T22:44:53.570 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:44:53.570 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:44:53.570 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.570 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.570 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_id": 1, 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: "rule_name": "test", 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: "ruleset": 1, 2020-03-11T22:44:53.571 INFO:tasks.workunit.client.0.smithi130.stdout: "type": 3, 2020-03-11T22:44:53.572 INFO:tasks.workunit.client.0.smithi130.stdout: "min_size": 3, 2020-03-11T22:44:53.572 INFO:tasks.workunit.client.0.smithi130.stdout: "max_size": 3, 2020-03-11T22:44:53.572 INFO:tasks.workunit.client.0.smithi130.stdout: "steps": [ 2020-03-11T22:44:53.572 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.572 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:44:53.572 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 5 2020-03-11T22:44:53.572 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.573 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.573 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "set_choose_tries", 2020-03-11T22:44:53.573 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 100 2020-03-11T22:44:53.573 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.573 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "take", 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: "item": -1, 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: "item_name": "default" 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "choose_indep", 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 0, 2020-03-11T22:44:53.574 INFO:tasks.workunit.client.0.smithi130.stdout: "type": "osd" 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "emit" 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.575 INFO:tasks.workunit.client.0.smithi130.stdout: "tunables": { 2020-03-11T22:44:53.576 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_tries": 0, 2020-03-11T22:44:53.576 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:44:53.576 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_total_tries": 50, 2020-03-11T22:44:53.577 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:44:53.577 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:44:53.577 INFO:tasks.workunit.client.0.smithi130.stdout: "chooseleaf_stable": 1, 2020-03-11T22:44:53.577 INFO:tasks.workunit.client.0.smithi130.stdout: "straw_calc_version": 1, 2020-03-11T22:44:53.577 INFO:tasks.workunit.client.0.smithi130.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:44:53.577 INFO:tasks.workunit.client.0.smithi130.stdout: "profile": "jewel", 2020-03-11T22:44:53.577 INFO:tasks.workunit.client.0.smithi130.stdout: "optimal_tunables": 1, 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "legacy_tunables": 0, 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "minimum_required_version": "jewel", 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables": 1, 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables2": 1, 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v2_rules": 1, 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables3": 1, 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v3_rules": 0, 2020-03-11T22:44:53.578 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v4_buckets": 1, 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: "require_feature_tunables5": 1, 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: "has_v5_rules": 0 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: "choose_args": {} 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: "fsmap": { 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:44:53.579 INFO:tasks.workunit.client.0.smithi130.stdout: "default_fscid": -1, 2020-03-11T22:44:53.580 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": { 2020-03-11T22:44:53.580 INFO:tasks.workunit.client.0.smithi130.stdout: "compat": {}, 2020-03-11T22:44:53.580 INFO:tasks.workunit.client.0.smithi130.stdout: "ro_compat": {}, 2020-03-11T22:44:53.580 INFO:tasks.workunit.client.0.smithi130.stdout: "incompat": { 2020-03-11T22:44:53.580 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_1": "base v0.20", 2020-03-11T22:44:53.580 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:44:53.580 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_8": "no anchor table", 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_9": "file layout v2", 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.581 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: "feature_flags": { 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: "enable_multiple": false, 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: "ever_enabled_multiple": false 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: "standbys": [], 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: "filesystems": [] 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.582 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:44:53.583 INFO:tasks.workunit.client.0.smithi130.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:44:53.583 INFO:tasks.workunit.client.0.smithi130.stdout: "auth": { 2020-03-11T22:44:53.583 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:44:53.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 5, 2020-03-11T22:44:53.583 INFO:tasks.workunit.client.0.smithi130.stdout: "num_secrets": 3 2020-03-11T22:44:53.583 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.583 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_active": 1, 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_unknown": 0, 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osd": 3, 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_sum": { 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48430, 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2020-03-11T22:44:53.584 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 1, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 303, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:53.585 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:44:53.586 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:44:53.586 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:44:53.586 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:44:53.586 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:44:53.586 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:53.586 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:53.587 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:53.587 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:53.587 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:53.587 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:53.587 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:44:53.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 0, 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 0, 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 18, 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 3, 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 3, 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 0 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum": { 2020-03-11T22:44:53.592 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:44:53.593 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:44:53.593 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:44:53.593 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:44:53.593 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:53.593 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:53.593 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:44:53.593 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3181120, 2020-03-11T22:44:53.594 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 35136, 2020-03-11T22:44:53.594 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 9, 2020-03-11T22:44:53.594 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145718, 2020-03-11T22:44:53.594 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311391680, 2020-03-11T22:44:53.594 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318865080320, 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 35979264, 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 1564375, 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:53.595 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:53.596 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 9284, 2020-03-11T22:44:53.596 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221216188 2020-03-11T22:44:53.596 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.596 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:44:53.596 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:53.596 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:53.596 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:53.597 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:44:53.597 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:44:53.597 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:44:53.597 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.598 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:44:53.598 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:53.598 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:53.598 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:53.598 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:44:53.598 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.598 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_sum_by_class": { 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: "hdd": { 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: "up_from": 0, 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 0, 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pgs": 3, 2020-03-11T22:44:53.599 INFO:tasks.workunit.client.0.smithi130.stdout: "num_osds": 3, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "kb": 314572800, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used": 3181120, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_data": 35136, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_omap": 9, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_used_meta": 3145718, 2020-03-11T22:44:53.600 INFO:tasks.workunit.client.0.smithi130.stdout: "kb_avail": 311391680, 2020-03-11T22:44:53.601 INFO:tasks.workunit.client.0.smithi130.stdout: "statfs": { 2020-03-11T22:44:53.601 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 322122547200, 2020-03-11T22:44:53.601 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 318865080320, 2020-03-11T22:44:53.601 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:53.601 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 35979264, 2020-03-11T22:44:53.601 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 1564375, 2020-03-11T22:44:53.601 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 9284, 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 3221216188 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: "hb_peers": [], 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:53.602 INFO:tasks.workunit.client.0.smithi130.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:53.603 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:53.603 INFO:tasks.workunit.client.0.smithi130.stdout: "op_queue_age_hist": { 2020-03-11T22:44:53.603 INFO:tasks.workunit.client.0.smithi130.stdout: "histogram": [], 2020-03-11T22:44:53.603 INFO:tasks.workunit.client.0.smithi130.stdout: "upper_bound": 1 2020-03-11T22:44:53.603 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.603 INFO:tasks.workunit.client.0.smithi130.stdout: "perf_stat": { 2020-03-11T22:44:53.603 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: "apply_latency_ns": 0 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: "alerts": [], 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: "network_ping_times": [] 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.604 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: "pool_stats": [ 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: "poolid": 1, 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg": 1, 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 48430, 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2020-03-11T22:44:53.605 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 1, 2020-03-11T22:44:53.606 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 303, 2020-03-11T22:44:53.606 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:53.606 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2020-03-11T22:44:53.606 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 101, 2020-03-11T22:44:53.606 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:53.606 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:53.606 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2020-03-11T22:44:53.607 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2020-03-11T22:44:53.607 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2020-03-11T22:44:53.607 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2020-03-11T22:44:53.607 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2020-03-11T22:44:53.607 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 101, 2020-03-11T22:44:53.607 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:53.608 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:53.608 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:53.608 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:53.608 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:53.608 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:53.608 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2020-03-11T22:44:53.609 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:53.610 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:53.610 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:53.610 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:53.610 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:53.610 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:53.610 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:53.610 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:53.611 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:53.611 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2020-03-11T22:44:53.611 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2020-03-11T22:44:53.611 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.611 INFO:tasks.workunit.client.0.smithi130.stdout: "store_stats": { 2020-03-11T22:44:53.611 INFO:tasks.workunit.client.0.smithi130.stdout: "total": 0, 2020-03-11T22:44:53.612 INFO:tasks.workunit.client.0.smithi130.stdout: "available": 0, 2020-03-11T22:44:53.612 INFO:tasks.workunit.client.0.smithi130.stdout: "internally_reserved": 0, 2020-03-11T22:44:53.612 INFO:tasks.workunit.client.0.smithi130.stdout: "allocated": 19857408, 2020-03-11T22:44:53.612 INFO:tasks.workunit.client.0.smithi130.stdout: "data_stored": 1241088, 2020-03-11T22:44:53.612 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed": 0, 2020-03-11T22:44:53.612 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:53.612 INFO:tasks.workunit.client.0.smithi130.stdout: "data_compressed_original": 0, 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_allocated": 0, 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: "internal_metadata": 0 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 18, 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 18, 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: "up": 3, 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": 3, 2020-03-11T22:44:53.613 INFO:tasks.workunit.client.0.smithi130.stdout: "num_store_stats": 3 2020-03-11T22:44:53.614 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.614 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.614 INFO:tasks.workunit.client.0.smithi130.stdout: "osd_stats": [ 2020-03-11T22:44:53.614 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.614 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:53.614 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 180388626442 2020-03-11T22:44:53.614 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.615 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.615 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:53.615 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 38654705682 2020-03-11T22:44:53.615 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.615 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.615 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: "seq": 193273528329 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_state": [ 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: "num": 1 2020-03-11T22:44:53.616 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.617 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.617 INFO:tasks.workunit.client.0.smithi130.stdout: "num_pg_by_osd": [ 2020-03-11T22:44:53.617 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.617 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2, 2020-03-11T22:44:53.617 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:44:53.617 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:44:53.618 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:53.618 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.618 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.618 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 2147483647, 2020-03-11T22:44:53.619 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:44:53.619 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:44:53.619 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:53.619 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.619 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.619 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 1, 2020-03-11T22:44:53.620 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 0, 2020-03-11T22:44:53.620 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 0, 2020-03-11T22:44:53.620 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:53.620 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.620 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.620 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": 0, 2020-03-11T22:44:53.621 INFO:tasks.workunit.client.0.smithi130.stdout: "num_primary_pg": 1, 2020-03-11T22:44:53.621 INFO:tasks.workunit.client.0.smithi130.stdout: "num_acting_pg": 1, 2020-03-11T22:44:53.621 INFO:tasks.workunit.client.0.smithi130.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:53.621 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.621 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.621 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [ 2020-03-11T22:44:53.621 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 1, 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": {} 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: "servicemap": { 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 1, 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: "modified": "2020-03-11T22:42:13.715486+0000", 2020-03-11T22:44:53.622 INFO:tasks.workunit.client.0.smithi130.stdout: "services": {} 2020-03-11T22:44:53.623 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:44:53.623 INFO:tasks.workunit.client.0.smithi130.stdout: "paxos": { 2020-03-11T22:44:53.623 INFO:tasks.workunit.client.0.smithi130.stdout: "first_committed": 1, 2020-03-11T22:44:53.623 INFO:tasks.workunit.client.0.smithi130.stdout: "last_committed": 443, 2020-03-11T22:44:53.623 INFO:tasks.workunit.client.0.smithi130.stdout: "last_pn": 0, 2020-03-11T22:44:53.623 INFO:tasks.workunit.client.0.smithi130.stdout: "accepted_pn": 0 2020-03-11T22:44:53.623 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:44:53.624 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:44:53.624 INFO:tasks.workunit.client.0.smithi130.stderr:report 203924945 2020-03-11T22:44:53.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:44:53.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:44:53.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:53.625 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery 2020-03-11T22:44:53.805 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:53.805 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:17.588938+0000 40'102 46:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:44:53.805 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:44:53.805 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:53.806 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:44:53.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:53.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:44:53.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:44:53.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:44:53.816 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:53.816 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:53.817 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:44:53.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.0.asok 2020-03-11T22:44:53.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:53.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:44:53.908 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:53.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:53.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:44:53.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:44:53.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:44:53.916 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:53.916 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:53.916 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:44:53.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.2.asok 2020-03-11T22:44:53.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:53.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:44:54.007 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:54.014 INFO:tasks.workunit.client.0.smithi130.stdout:revive divergent 1 2020-03-11T22:44:54.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:44:54.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:54.334 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:54.501 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:17.588938+0000 40'102 46:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:44:54.501 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:44:54.502 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:54.502 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:44:54.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:44:55.058 INFO:tasks.workunit.client.0.smithi130.stderr:noup is set 2020-03-11T22:44:55.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:44:55.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:55.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:55.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:44:55.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:55.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:44:55.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:55.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:55.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:55.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:55.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:44:55.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:44:55.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:55.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:55.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:55.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:55.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:55.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:55.072 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:55.072 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:55.072 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:44:55.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:44:55.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:44:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:55.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:55.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:44:55.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:44:55.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:55.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:44:55.074 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:44:55.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:44:55.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:55.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:55.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:55.095 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:55.092+0000 7f85260dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:55.097 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:55.094+0000 7f85260dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:55.098 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:55.095+0000 7f85260dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:55.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:44:55.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:44:55.617 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:55.611+0000 7f85260dcec0 -1 Falling back to public interface 2020-03-11T22:44:56.449 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:44:56.445+0000 7f85260dcec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:45:00.433 INFO:tasks.workunit.client.0.smithi130.stdout:delay recovery divergent 2020-03-11T22:45:00.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:45:00.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:00.763 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:00.763 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:44:17.588938+0000 40'102 46:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:45:00.763 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:45:00.763 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:00.764 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:45:00.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:45:00.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:45:00.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:45:00.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:00.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:00.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:45:00.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17424/ceph-osd.1.asok 2020-03-11T22:45:00.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:45:00.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17424/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:45:00.865 INFO:tasks.workunit.client.0.smithi130.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:45:00.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:45:01.595 INFO:tasks.workunit.client.0.smithi130.stderr:noup is unset 2020-03-11T22:45:01.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:45:01.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:01.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:01.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:01.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:01.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:01.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:01.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:01.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:01.605 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:01.975 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 42 up_thru 45 down_at 41 last_clean_interval [34,40) [v2:127.0.0.1:6802/50310,v1:127.0.0.1:6803/50310] [v2:127.0.0.1:6804/50310,v1:127.0.0.1:6805/50310] exists,up d34a39d8-760e-4854-8c86-11e878da47ec 2020-03-11T22:45:01.975 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:01.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:01.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:01.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:01.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:45:01.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:01.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:01.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:01.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:01.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:01.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:01.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:01.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:02.316 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/51957,v1:127.0.0.1:6819/51957] [v2:127.0.0.1:6820/51957,v1:127.0.0.1:6821/51957] exists,up 05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5 2020-03-11T22:45:02.317 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:02.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:02.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:02.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:02.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:45:02.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:02.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:02.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:02.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:02.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:02.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:02.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:02.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:02.683 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 45 up_thru 0 down_at 44 last_clean_interval [35,40) [v2:127.0.0.1:6810/50542,v1:127.0.0.1:6811/50542] [v2:127.0.0.1:6812/50542,v1:127.0.0.1:6813/50542] exists,up 9bd7863a-9ae2-4649-8393-d7b70b4864ee 2020-03-11T22:45:02.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:02.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:02.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:02.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:03.004 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:03.004 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 peering 2020-03-11T22:45:01.416145+0000 28'104 50:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:45:03.004 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:45:03.005 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:03.006 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:45:03.014 INFO:tasks.workunit.client.0.smithi130.stdout:wait for peering 2020-03-11T22:45:03.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:45:03.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:03.332 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:03.333 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 peering 2020-03-11T22:45:01.416145+0000 28'104 50:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:45:03.333 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:45:03.333 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:03.334 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:45:03.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:45:13.612 INFO:tasks.workunit.client.0.smithi130.stdout:killing divergent 1 2020-03-11T22:45:13.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:45:13.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:13.943 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:13.943 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:45:01.708500+0000 40'102 51:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:45:13.943 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:45:13.944 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:13.945 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:45:13.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:45:13.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:13.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:13.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:13.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:13.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:45:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:14.059 INFO:tasks.workunit.client.0.smithi130.stdout:reviving divergent 1 2020-03-11T22:45:14.379 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:14.379 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:45:01.708500+0000 40'102 51:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:45:14.379 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:45:14.379 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:14.381 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:45:14.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:45:14.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:14.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:14.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:45:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:45:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:14.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:45:14.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:45:14.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:14.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:14.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:14.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:14.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:14.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:45:14.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:45:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' 2020-03-11T22:45:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:14.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:14.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:14.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:14.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:14.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:14.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:14.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:14.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:45:14.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:45:14.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=70af20e8-df52-4c90-ac53-59b9c01e7b62 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17424/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:14.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:45:14.397 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:45:14.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:45:14.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:14.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:14.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:14.418 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:14.415+0000 7f457a2c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:14.437 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:14.434+0000 7f457a2c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:14.438 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:14.435+0000 7f457a2c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:14.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:45:14.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:14.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:14.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:14.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:14.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:14.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:14.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:14.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:14.773 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:14.954 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:14.951+0000 7f457a2c2ec0 -1 Falling back to public interface 2020-03-11T22:45:15.119 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/51957,v1:127.0.0.1:6819/51957] [v2:127.0.0.1:6820/51957,v1:127.0.0.1:6821/51957] exists,up 05a5f37e-ffc3-42d0-ab34-6faaa9ebe3a5 2020-03-11T22:45:15.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:15.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:15.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:15.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:45:15.756 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:15.753+0000 7f457a2c2ec0 -1 osd.1 51 log_to_monitors {default=true} 2020-03-11T22:45:35.120 INFO:tasks.workunit.client.0.smithi130.stdout:allowing recovery 2020-03-11T22:45:35.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:45:35.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:35.456 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:35.456 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:45:16.972450+0000 51'103 54:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:45:35.456 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:45:35.457 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:35.458 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:45:35.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:35.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:45:35.586 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:35.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:35.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:45:35.714 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:45:35.843 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:35.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:45:35.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:35.851 INFO:tasks.workunit.client.0.smithi130.stdout:reading divergent objects 2020-03-11T22:45:36.169 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:36.169 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:45:16.972450+0000 51'103 54:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.054968+0000 0'0 2020-03-11T22:42:36.054968+0000 0 2020-03-11T22:45:36.169 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:45:36.169 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:36.170 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:45:36.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:45:36.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:45:36.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:36.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:45:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:45:36.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:36.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:45:36.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:36.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:45:36.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:45:36.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:36.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:36.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:42:56.654201+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 20 _merge_object_divergent_entries: keeping 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:42:56.654201+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:36.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 27'99 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:45:36.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:45:36.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:45:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:43:03.216785+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 20 _merge_object_divergent_entries: keeping 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:43:03.216785+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 27'99 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:45:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:45:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 28'104 2020-03-11T22:45:36.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:43:03.216785+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:45:36.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-03-11T22:43:02.054015+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:43:03.058009+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-03-11T22:43:02.054015+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 20 _merge_object_divergent_entries: keeping 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:43:03.058009+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'102 2020-03-11T22:45:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:45:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'104 2020-03-11T22:45:36.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:43:03.058009+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:36.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:01.657+0000 7f8504297700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:45:36.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:36.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:36.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:45:36.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:45:36.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:45:36.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:45:36.327 INFO:tasks.workunit.client.0.smithi130.stdout:success 2020-03-11T22:45:36.793 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' does not exist 2020-03-11T22:45:36.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:45:36.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:36.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:36.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:36.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:36.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:36.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:36.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:45:36.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:36.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:36.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:36.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:36.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:36.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:36.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:36.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:36.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:36.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:36.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:36.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:36.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:36.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:36.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:36.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:36.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:36.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:36.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:36.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:36.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:36.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:36.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:36.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:36.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:45:36.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:45:36.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:36.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:36.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:45:36.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:45:36.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:36.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:36.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:36.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:36.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:36.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:36.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:36.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:36.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:36.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:36.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:36.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:36.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:36.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:45:36.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:36.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:36.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:36.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17424 2020-03-11T22:45:36.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17424 2020-03-11T22:45:36.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:36.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:36.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:45:36.946 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:45:36.947 INFO:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:45:37.014 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:45:37.014 INFO:teuthology.orchestra.run.smithi130:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:45:37.044 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:45:37.052 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:45:37.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:37.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:37.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:45:37.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:45:37.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:45:37.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:45:37.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:45:37.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:45:37.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:45:37.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:45:37.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:45:37.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:45:37.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:45:37.054 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:45:37.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none ' 2020-03-11T22:45:37.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:45:37.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:45:37.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:45:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:45:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:45:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:45:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:45:37.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:37.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:37.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:37.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:37.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:37.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:37.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:37.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:37.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:37.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:37.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:37.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:37.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:37.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:37.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:45:37.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:37.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:37.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:37.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52885 2020-03-11T22:45:37.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:37.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:37.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:45:37.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:45:37.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:37.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:37.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52885 2020-03-11T22:45:37.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:45:37.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:45:37.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:45:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:45:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:45:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:45:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:45:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:45:37.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:45:37.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:45:37.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:45:37.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:45:37.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:45:37.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:37.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:37.391 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:37.391 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:37.392 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:37.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:37.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:45:37.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:45:37.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:45:37.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:37.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:37.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:45:37.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:45:37.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:37.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:37.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:37.553 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:37.553 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:37.553 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:37.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52885/ceph-mon.a.asok 2020-03-11T22:45:37.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:37.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52885/ceph-mon.a.asok config get fsid 2020-03-11T22:45:37.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:45:37.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:37.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:37.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:45:37.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:45:37.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:37.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:37.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:37.639 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:37.640 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:37.640 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:37.640 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52885/ceph-mon.a.asok 2020-03-11T22:45:37.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:37.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52885/ceph-mon.a.asok config get mon_host 2020-03-11T22:45:37.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:45:37.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:45:37.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:45:37.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:45:37.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:45:37.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:45:37.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:45:38.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:45:38.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:38.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:38.592 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:38.592 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:38.592 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:38.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:38.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:38.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:45:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:45:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:45:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:38.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:38.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:38.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:45:38.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:45:38.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:38.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:38.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:38.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:38.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:38.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:38.617 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:38.617 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:38.617 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:38.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:38.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:38.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:38.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:38.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:38.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:38.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:38.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:38.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:38.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:38.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:38.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:38.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:45:38.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:38.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d91969d1-56d1-4c28-8d18-418ec9acbd83 2020-03-11T22:45:38.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 d91969d1-56d1-4c28-8d18-418ec9acbd83' 2020-03-11T22:45:38.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:38.621 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 d91969d1-56d1-4c28-8d18-418ec9acbd83 2020-03-11T22:45:38.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQASamlezcWnJRAAhWn0kr+zrwUi9IPmbrB6CA== 2020-03-11T22:45:38.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQASamlezcWnJRAAhWn0kr+zrwUi9IPmbrB6CA=="}' 2020-03-11T22:45:38.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d91969d1-56d1-4c28-8d18-418ec9acbd83 -i td/ec-error-rollforward/0/new.json 2020-03-11T22:45:39.004 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:39.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:45:39.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQASamlezcWnJRAAhWn0kr+zrwUi9IPmbrB6CA== --osd-uuid d91969d1-56d1-4c28-8d18-418ec9acbd83 2020-03-11T22:45:39.034 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:39.030+0000 7f433736bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:39.035 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:39.032+0000 7f433736bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:39.036 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:39.033+0000 7f433736bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:39.037 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:39.033+0000 7f433736bec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:39.037 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:39.033+0000 7f433736bec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:39.037 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:39.033+0000 7f433736bec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:45:39.066 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:39.062+0000 7f433736bec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:45:42.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:45:42.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:42.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:45:42.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:42.088 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:45:42.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:45:42.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:42.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:42.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:42.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:42.534 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:45:42.551 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:42.548+0000 7f4697b84ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:42.577 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:42.574+0000 7f4697b84ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:42.578 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:42.575+0000 7f4697b84ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:42.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:45:42.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:42.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:42.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:42.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:42.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:42.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:42.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:42.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:42.890 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:43.095 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:43.091+0000 7f4697b84ec0 -1 Falling back to public interface 2020-03-11T22:45:43.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:43.834 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:43.830+0000 7f4697b84ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:45:44.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:44.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:44.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:44.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:44.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:44.233 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:45:44.581 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53314,v1:127.0.0.1:6803/53314] [v2:127.0.0.1:6804/53314,v1:127.0.0.1:6805/53314] exists,up d91969d1-56d1-4c28-8d18-418ec9acbd83 2020-03-11T22:45:44.581 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 3cf88988-9d15-43d5-b455-05b584719bc8 2020-03-11T22:45:44.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:44.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:44.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:44.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:45:44.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:44.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:44.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:45:44.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:44.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:45:44.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:44.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:44.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:44.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:44.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:45:44.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:45:44.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:44.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:44.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:44.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:44.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:44.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:44.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:44.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:44.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:45:44.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:44.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3cf88988-9d15-43d5-b455-05b584719bc8 2020-03-11T22:45:44.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 3cf88988-9d15-43d5-b455-05b584719bc8' 2020-03-11T22:45:44.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:44.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAYamleEkMtIxAAi03uvTbObzRK7saiTUPjgQ== 2020-03-11T22:45:44.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAYamleEkMtIxAAi03uvTbObzRK7saiTUPjgQ=="}' 2020-03-11T22:45:44.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3cf88988-9d15-43d5-b455-05b584719bc8 -i td/ec-error-rollforward/1/new.json 2020-03-11T22:45:45.087 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:45:45.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:45:45.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAYamleEkMtIxAAi03uvTbObzRK7saiTUPjgQ== --osd-uuid 3cf88988-9d15-43d5-b455-05b584719bc8 2020-03-11T22:45:45.117 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:45.114+0000 7fe9cadb2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:45.119 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:45.116+0000 7fe9cadb2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:45.119 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:45.116+0000 7fe9cadb2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:45.120 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:45.117+0000 7fe9cadb2ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:45.120 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:45.117+0000 7fe9cadb2ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:45.120 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:45.117+0000 7fe9cadb2ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:45:45.190 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:45.187+0000 7fe9cadb2ec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:45:48.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:45:48.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:48.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:45:48.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:48.155 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:45:48.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:45:48.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:48.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:48.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:48.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:48.609 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:45:48.626 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:48.622+0000 7f8e39d87ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:48.627 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:48.624+0000 7f8e39d87ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:48.628 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:48.625+0000 7f8e39d87ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:48.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:45:48.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:48.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:48.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:48.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:48.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:48.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:48.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:48.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:48.949 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:49.143 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:49.140+0000 7f8e39d87ec0 -1 Falling back to public interface 2020-03-11T22:45:49.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:49.926 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:49.923+0000 7f8e39d87ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:45:50.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:50.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:50.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:50.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:50.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:50.297 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:45:50.661 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53669,v1:127.0.0.1:6811/53669] [v2:127.0.0.1:6812/53669,v1:127.0.0.1:6813/53669] exists,up 3cf88988-9d15-43d5-b455-05b584719bc8 2020-03-11T22:45:50.661 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 353c8a37-4e4c-4df0-9ab3-7ff22f5f81e3 2020-03-11T22:45:50.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:50.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:50.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:50.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:45:50.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:50.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:50.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:45:50.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:50.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:45:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:45:50.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:45:50.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:50.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:50.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:50.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:50.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:50.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:50.666 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:50.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:50.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:50.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:50.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:50.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:50.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:50.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:50.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:50.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:50.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:50.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:50.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:50.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:50.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:50.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:45:50.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:50.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=353c8a37-4e4c-4df0-9ab3-7ff22f5f81e3 2020-03-11T22:45:50.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 353c8a37-4e4c-4df0-9ab3-7ff22f5f81e3' 2020-03-11T22:45:50.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:50.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAeamleC9b0JxAAMqGnywfLiM+G/YviolNIdw== 2020-03-11T22:45:50.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAeamleC9b0JxAAMqGnywfLiM+G/YviolNIdw=="}' 2020-03-11T22:45:50.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 353c8a37-4e4c-4df0-9ab3-7ff22f5f81e3 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:45:51.163 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:45:51.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:45:51.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAeamleC9b0JxAAMqGnywfLiM+G/YviolNIdw== --osd-uuid 353c8a37-4e4c-4df0-9ab3-7ff22f5f81e3 2020-03-11T22:45:51.194 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:51.190+0000 7febebfecec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:51.221 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:51.218+0000 7febebfecec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:51.223 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:51.220+0000 7febebfecec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:51.224 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:51.220+0000 7febebfecec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:51.224 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:51.220+0000 7febebfecec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:51.224 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:51.220+0000 7febebfecec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:45:51.288 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:51.286+0000 7febebfecec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:45:54.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:45:54.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:54.307 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:45:54.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:45:54.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:54.758 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:45:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:45:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:54.776 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:54.774+0000 7fba14cfbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:54.778 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:54.776+0000 7fba14cfbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:54.780 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:54.778+0000 7fba14cfbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:55.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:45:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:55.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:55.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:55.113 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:45:55.297 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:55.294+0000 7fba14cfbec0 -1 Falling back to public interface 2020-03-11T22:45:55.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:56.046 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:56.044+0000 7fba14cfbec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:45:56.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:56.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:56.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:56.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:56.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:56.455 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:45:56.802 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54024,v1:127.0.0.1:6819/54024] [v2:127.0.0.1:6820/54024,v1:127.0.0.1:6821/54024] exists,up 353c8a37-4e4c-4df0-9ab3-7ff22f5f81e3 2020-03-11T22:45:56.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:56.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:56.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:56.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:45:56.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:56.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:56.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:45:56.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:56.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:45:56.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:56.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:56.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:56.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:56.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:45:56.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:45:56.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:56.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:56.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:56.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:56.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:56.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:56.805 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:56.805 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:56.805 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:45:56.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:56.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:45:56.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:56.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:56.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:56.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:56.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:56.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:56.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:56.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:56.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:56.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:56.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:45:56.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:56.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=005a4309-b783-4732-b865-f958cc4fb8c2 2020-03-11T22:45:56.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 005a4309-b783-4732-b865-f958cc4fb8c2' 2020-03-11T22:45:56.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:56.809 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 005a4309-b783-4732-b865-f958cc4fb8c2 2020-03-11T22:45:56.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAkamleHLKBMBAAYAd1quI4JEak9wmKhJkrdQ== 2020-03-11T22:45:56.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAkamleHLKBMBAAYAd1quI4JEak9wmKhJkrdQ=="}' 2020-03-11T22:45:56.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 005a4309-b783-4732-b865-f958cc4fb8c2 -i td/ec-error-rollforward/3/new.json 2020-03-11T22:45:57.352 INFO:tasks.workunit.client.0.smithi130.stdout:3 2020-03-11T22:45:57.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:45:57.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAkamleHLKBMBAAYAd1quI4JEak9wmKhJkrdQ== --osd-uuid 005a4309-b783-4732-b865-f958cc4fb8c2 2020-03-11T22:45:57.382 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:57.380+0000 7f871b917ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:57.381+0000 7f871b917ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:57.385 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:57.383+0000 7f871b917ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:57.386 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:57.383+0000 7f871b917ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:57.386 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:57.383+0000 7f871b917ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:57.386 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:57.383+0000 7f871b917ec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:45:57.464 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:45:57.462+0000 7f871b917ec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:46:00.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:46:00.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:00.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:46:00.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:00.417 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2020-03-11T22:46:00.872 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2020-03-11T22:46:00.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:46:00.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:00.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:00.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:00.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:00.890 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:00.887+0000 7f3ecf648ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:00.891 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:00.889+0000 7f3ecf648ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:00.893 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:00.891+0000 7f3ecf648ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:01.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:46:01.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:01.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:46:01.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:01.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:01.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:01.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:01.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:01.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:01.236 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:01.413 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:01.410+0000 7f3ecf648ec0 -1 Falling back to public interface 2020-03-11T22:46:01.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:02.248 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:02.246+0000 7f3ecf648ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:02.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:02.576 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:46:02.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:03.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:03.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:03.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:46:03.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:03.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:03.946 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:46:04.296 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54379,v1:127.0.0.1:6827/54379] [v2:127.0.0.1:6828/54379,v1:127.0.0.1:6829/54379] exists,up 005a4309-b783-4732-b865-f958cc4fb8c2 2020-03-11T22:46:04.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:04.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:04.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:04.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:46:04.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:46:05.610 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'ec' already exists 2020-03-11T22:46:05.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:46:06.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:46:06.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 54024 2020-03-11T22:46:06.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=54662 2020-03-11T22:46:06.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:46:06.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:46:07.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 54685' 2020-03-11T22:46:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:46:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 54686' 2020-03-11T22:46:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:46:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 54687' 2020-03-11T22:46:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:46:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:46:08.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:46:08.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:08.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:08.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:08.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:08.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:08.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:08.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 54662 54685 54686 54687 2020-03-11T22:46:08.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:46:08.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:46:08.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:08.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:08.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:46:08.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:08.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:46:08.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:08.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:08.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:08.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:08.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:46:08.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:46:08.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:08.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:08.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:08.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:08.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:08.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:46:08.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:46:08.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:08.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:46:08.886 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:46:08.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:46:08.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:08.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:08.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:08.901 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:08.897+0000 7f96fa98cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:08.903 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:08.900+0000 7f96fa98cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:08.904 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:08.901+0000 7f96fa98cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:09.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:46:09.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:09.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:09.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:09.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:09.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:09.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:09.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:09.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:09.229 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:09.420 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:09.417+0000 7f96fa98cec0 -1 Falling back to public interface 2020-03-11T22:46:09.573 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53314,v1:127.0.0.1:6803/53314] [v2:127.0.0.1:6804/53314,v1:127.0.0.1:6805/53314] exists,up d91969d1-56d1-4c28-8d18-418ec9acbd83 2020-03-11T22:46:09.573 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:46:09.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:09.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:09.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:09.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:46:09.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:46:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:46:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:09.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:09.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:46:09.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:46:09.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:09.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:09.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:09.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:09.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:09.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:09.577 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:09.577 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:09.577 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:46:09.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:09.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:09.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:09.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:09.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:09.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:09.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:09.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:09.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:09.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:09.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:46:09.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:46:09.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:09.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:46:09.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:46:09.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:09.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:09.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:09.589 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:09.586+0000 7f02e73e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:09.591 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:09.588+0000 7f02e73e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:09.592 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:09.589+0000 7f02e73e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:46:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:09.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:09.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:09.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:09.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:09.919 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:10.118 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:10.105+0000 7f02e73e4ec0 -1 Falling back to public interface 2020-03-11T22:46:10.267 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53669,v1:127.0.0.1:6811/53669] [v2:127.0.0.1:6812/53669,v1:127.0.0.1:6813/53669] exists,up 3cf88988-9d15-43d5-b455-05b584719bc8 2020-03-11T22:46:10.267 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:46:10.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:10.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:10.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:10.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:46:10.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:10.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:10.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:46:10.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:10.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:46:10.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:10.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:10.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:10.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:10.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:46:10.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:46:10.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:10.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:10.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:10.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:10.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:10.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:10.270 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:10.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:10.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:10.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:10.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:10.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:10.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:46:10.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:46:10.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:10.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:46:10.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:46:10.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:10.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:10.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:10.284 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:10.281+0000 7f2561f0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:10.286 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:10.282+0000 7f2561f0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:10.287 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:10.284+0000 7f2561f0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:10.315 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:10.312+0000 7f96fa98cec0 -1 osd.0 26 log_to_monitors {default=true} 2020-03-11T22:46:10.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:46:10.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:10.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:46:10.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:10.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:10.616 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:10.803 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:10.799+0000 7f2561f0aec0 -1 Falling back to public interface 2020-03-11T22:46:11.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:11.183 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:11.180+0000 7f02e73e4ec0 -1 osd.1 26 log_to_monitors {default=true} 2020-03-11T22:46:12.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:12.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:12.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:12.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:12.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:12.052 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:46:12.163 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:12.159+0000 7f2561f0aec0 -1 osd.2 25 log_to_monitors {default=true} 2020-03-11T22:46:12.444 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [15,25) [v2:127.0.0.1:6818/55020,v1:127.0.0.1:6819/55020] [v2:127.0.0.1:6820/55020,v1:127.0.0.1:6821/55020] exists,up 353c8a37-4e4c-4df0-9ab3-7ff22f5f81e3 2020-03-11T22:46:12.444 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2020-03-11T22:46:12.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:12.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:12.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:12.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:46:12.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:12.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:12.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:46:12.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:12.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:46:12.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:12.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:12.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' 2020-03-11T22:46:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:12.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:46:12.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:46:12.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=53dcc28e-a8f7-49f0-9fb7-41bd9079a500 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52885/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:12.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:46:12.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:46:12.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:12.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:12.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:12.461 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:12.458+0000 7f4dbf4e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:12.463 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:12.460+0000 7f4dbf4e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:12.465 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:12.461+0000 7f4dbf4e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:46:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:46:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:12.804 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:12.980 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:12.977+0000 7f4dbf4e6ec0 -1 Falling back to public interface 2020-03-11T22:46:13.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:13.861 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:13.857+0000 7f4dbf4e6ec0 -1 osd.3 26 log_to_monitors {default=true} 2020-03-11T22:46:14.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:14.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:14.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:14.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:14.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:46:14.145 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:46:14.543 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 35 up_thru 0 down_at 29 last_clean_interval [20,26) [v2:127.0.0.1:6826/55326,v1:127.0.0.1:6827/55326] [v2:127.0.0.1:6828/55326,v1:127.0.0.1:6829/55326] exists,up 005a4309-b783-4732-b865-f958cc4fb8c2 2020-03-11T22:46:14.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:14.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:14.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:14.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:46:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:14.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:14.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:14.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:14.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:14.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:14.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:14.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:14.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:14.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:14.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:14.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:14.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:14.973 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:46:14.973 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:46:14.974 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2020-03-11T22:46:14.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:14.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:14.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=120259084291 2020-03-11T22:46:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291' 2020-03-11T22:46:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:15.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:15.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=133143986179 2020-03-11T22:46:15.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291 1-133143986179' 2020-03-11T22:46:15.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:15.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:15.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:46:15.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-141733920771' 2020-03-11T22:46:15.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:15.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:46:15.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:46:15.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-141733920771 3-150323855363' 2020-03-11T22:46:15.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:15.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-120259084291 2020-03-11T22:46:15.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:15.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:15.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-120259084291 2020-03-11T22:46:15.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:15.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=120259084291 2020-03-11T22:46:15.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2020-03-11T22:46:15.461 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 120259084291 2020-03-11T22:46:15.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:15.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 120259084291 2020-03-11T22:46:15.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:16.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:16.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:17.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 120259084292 -lt 120259084291 2020-03-11T22:46:17.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:17.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-133143986179 2020-03-11T22:46:17.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:17.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:17.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-133143986179 2020-03-11T22:46:17.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:17.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=133143986179 2020-03-11T22:46:17.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2020-03-11T22:46:17.156 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 133143986179 2020-03-11T22:46:17.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:17.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 133143986179 -lt 133143986179 2020-03-11T22:46:17.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:17.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-141733920771 2020-03-11T22:46:17.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:17.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:17.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-141733920771 2020-03-11T22:46:17.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:17.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:46:17.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2020-03-11T22:46:17.501 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 141733920771 2020-03-11T22:46:17.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:17.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920771 -lt 141733920771 2020-03-11T22:46:17.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:17.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-150323855363 2020-03-11T22:46:17.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:17.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:46:17.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-150323855363 2020-03-11T22:46:17.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:17.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:46:17.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2020-03-11T22:46:17.851 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 150323855363 2020-03-11T22:46:17.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:46:18.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:46:18.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:18.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:18.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:18.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:46:18.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:18.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:18.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:18.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:18.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:18.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:18.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:18.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:46:18.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:18.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:18.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:19.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:46:19.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:19.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:19.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:46:19.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:19.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:19.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:19.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:19.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:19.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:19.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:19.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:19.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:19.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:19.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:19.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:19.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:19.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:19.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:19.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:19.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:19.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:19.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:19.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:19.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:19.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52885 2020-03-11T22:46:19.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:19.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:19.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:46:19.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:46:19.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:19.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:46:19.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:19.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:19.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:19.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:19.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:19.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:19.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:19.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:19.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:19.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:19.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:19.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:19.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:19.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:19.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:19.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:19.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:46:19.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:19.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:19.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:19.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52885 2020-03-11T22:46:19.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52885 2020-03-11T22:46:19.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:19.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:19.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:46:19.571 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:46:19.571 INFO:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:46:19.644 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:46:19.645 INFO:teuthology.orchestra.run.smithi130:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:46:19.696 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:46:19.710 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:46:19.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:19.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:19.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:46:19.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:46:19.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:46:19.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:46:19.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:46:19.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:46:19.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:46:19.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:46:19.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:46:19.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:46:19.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:46:19.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:46:19.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none ' 2020-03-11T22:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:46:19.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:46:19.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:46:19.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:46:19.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:46:19.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:46:19.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:46:19.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:46:19.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:46:19.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:46:19.715 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_priority' 2020-03-11T22:46:19.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:46:19.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:46:19.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:46:19.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:46:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:46:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:46:19.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:19.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:19.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:19.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:19.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:19.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:19.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:19.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:19.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:19.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:46:19.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:19.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:19.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56067 2020-03-11T22:46:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:46:19.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:46:19.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:19.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:19.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56067 2020-03-11T22:46:19.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:46:19.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:46:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:46:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:46:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:46:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:46:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:46:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:46:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:46:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:46:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:46:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:46:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:46:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:46:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:46:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:46:20.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:46:20.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:20.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:20.021 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:20.021 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:20.021 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:20.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:46:20.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:46:20.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:46:20.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:46:20.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:20.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:20.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:46:20.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:46:20.174 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:20.174 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:20.174 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:20.174 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:20.174 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:20.174 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-mon.a.asok 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56067/ceph-mon.a.asok config get fsid 2020-03-11T22:46:20.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:46:20.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:20.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:20.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:46:20.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:46:20.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:20.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:20.262 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:20.262 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:20.262 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:20.262 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:20.262 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-mon.a.asok 2020-03-11T22:46:20.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:20.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56067/ceph-mon.a.asok config get mon_host 2020-03-11T22:46:20.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:46:20.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:46:20.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:46:20.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:46:20.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:46:20.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:46:20.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:46:20.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:46:20.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:20.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:20.820 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:20.820 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:20.820 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:20.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:46:20.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:20.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:20.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:46:20.843 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:46:20.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:46:20.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:46:20.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:46:20.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:46:20.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:20.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:46:20.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:20.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:46:20.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:20.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:20.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:20.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:20.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:46:20.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:46:20.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:20.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:20.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:46:20.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:46:20.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:20.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:20.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:20.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:46:20.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:46:20.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:20.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:20.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:20.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:20.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:20.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:46:20.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:20.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2a18ce39-c409-4f7b-ae25-b5b8147fc25c 2020-03-11T22:46:20.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 2a18ce39-c409-4f7b-ae25-b5b8147fc25c' 2020-03-11T22:46:20.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:20.853 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 2a18ce39-c409-4f7b-ae25-b5b8147fc25c 2020-03-11T22:46:20.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA8amlekyT+MxAADy2lJlh5JKVrt5HX0d2uOw== 2020-03-11T22:46:20.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA8amlekyT+MxAADy2lJlh5JKVrt5HX0d2uOw=="}' 2020-03-11T22:46:20.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2a18ce39-c409-4f7b-ae25-b5b8147fc25c -i td/osd-backfill-prio/0/new.json 2020-03-11T22:46:21.211 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:21.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:46:21.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8amlekyT+MxAADy2lJlh5JKVrt5HX0d2uOw== --osd-uuid 2a18ce39-c409-4f7b-ae25-b5b8147fc25c 2020-03-11T22:46:21.247 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:21.243+0000 7fa511eefec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.248 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:21.245+0000 7fa511eefec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.250 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:21.247+0000 7fa511eefec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.251 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:21.247+0000 7fa511eefec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:21.251 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:21.247+0000 7fa511eefec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:21.251 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:21.247+0000 7fa511eefec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:21.295 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:21.291+0000 7fa511eefec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:46:24.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:46:24.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:24.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:46:24.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:24.304 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:46:24.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:46:24.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:24.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:24.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:24.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:24.754 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:46:24.776 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:24.773+0000 7ff21e098ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:24.778 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:24.775+0000 7ff21e098ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:24.779 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:24.776+0000 7ff21e098ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:46:25.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:25.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:25.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:25.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:25.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:25.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:25.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:25.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:25.101 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:25.296 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:25.293+0000 7ff21e098ec0 -1 Falling back to public interface 2020-03-11T22:46:25.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:26.003 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:26.000+0000 7ff21e098ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:46:26.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:26.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:26.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:26.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:26.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:26.461 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:46:26.876 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56499,v1:127.0.0.1:6803/56499] [v2:127.0.0.1:6804/56499,v1:127.0.0.1:6805/56499] exists,up 2a18ce39-c409-4f7b-ae25-b5b8147fc25c 2020-03-11T22:46:26.876 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 be0a76b4-a0a8-4b96-bcfe-668b197caad6 2020-03-11T22:46:26.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:26.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:46:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:46:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:46:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:46:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:46:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:26.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:46:26.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:46:26.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:26.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:26.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:46:26.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:26.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:26.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:26.880 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:26.880 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:26.880 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:46:26.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:46:26.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:46:26.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:26.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:26.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:46:26.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:26.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=be0a76b4-a0a8-4b96-bcfe-668b197caad6 2020-03-11T22:46:26.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 be0a76b4-a0a8-4b96-bcfe-668b197caad6' 2020-03-11T22:46:26.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:26.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBCamleq1L7NBAAYBy5J7epwcmi58IeFfcuMw== 2020-03-11T22:46:26.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBCamleq1L7NBAAYBy5J7epwcmi58IeFfcuMw=="}' 2020-03-11T22:46:26.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new be0a76b4-a0a8-4b96-bcfe-668b197caad6 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:46:27.348 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:46:27.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:46:27.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBCamleq1L7NBAAYBy5J7epwcmi58IeFfcuMw== --osd-uuid be0a76b4-a0a8-4b96-bcfe-668b197caad6 2020-03-11T22:46:27.384 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:27.380+0000 7f0d7e3d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:27.401 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:27.398+0000 7f0d7e3d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:27.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:27.399+0000 7f0d7e3d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:27.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:27.399+0000 7f0d7e3d9ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:27.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:27.399+0000 7f0d7e3d9ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:27.403 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:27.399+0000 7f0d7e3d9ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:46:27.493 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:27.490+0000 7f0d7e3d9ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:46:30.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:46:30.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:30.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:46:30.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:30.521 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:46:30.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:46:30.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:30.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:30.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:30.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:30.980 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:46:31.002 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:30.999+0000 7f55d3a97ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:31.004 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:31.000+0000 7f55d3a97ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:31.005 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:31.002+0000 7f55d3a97ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:31.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:46:31.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:31.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:31.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:31.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:31.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:31.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:31.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:31.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:31.324 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:46:31.521 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:31.517+0000 7f55d3a97ec0 -1 Falling back to public interface 2020-03-11T22:46:31.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:32.229 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:46:32.225+0000 7f55d3a97ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:46:32.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:32.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:32.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:32.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:32.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:32.668 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:46:33.019 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56854,v1:127.0.0.1:6811/56854] [v2:127.0.0.1:6812/56854,v1:127.0.0.1:6813/56854] exists,up be0a76b4-a0a8-4b96-bcfe-668b197caad6 2020-03-11T22:46:33.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:33.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:33.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:33.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:33.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:33.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:46:33.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:46:33.598 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' already exists 2020-03-11T22:46:33.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:34.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:46:35.204 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 2 2020-03-11T22:46:35.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:35.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:46:35.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:46:35.823 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' already exists 2020-03-11T22:46:35.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:36.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:46:37.533 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 2 2020-03-11T22:46:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:46:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:46:38.014 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' already exists 2020-03-11T22:46:38.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:39.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:46:39.754 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 2 2020-03-11T22:46:39.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:46:44.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:46:44.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:44.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:44.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:44.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:44.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:44.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:44.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:44.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:44.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:44.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:44.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:44.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:44.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:44.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:44.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:45.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:45.216 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2020-03-11T22:46:45.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:45.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:45.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:45.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:46:45.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:46:45.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:45.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:45.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:46:45.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:46:45.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:45.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:45.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:45.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:45.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:45.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:45.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:46:45.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:46:45.506 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:46:45.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:45.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:46:45.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:46.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:46.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:47.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:46:47.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:47.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:46:47.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:47.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:47.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:46:47.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:47.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:46:47.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:46:47.209 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:46:47.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:47.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:46:47.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:47.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:47.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:48.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:46:48.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:48.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:48.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:48.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:48.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:48.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:48.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:48.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:46:48.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:48.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:48.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:48.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:46:48.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:48.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:48.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:46:49.124 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:49.124 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:39.974545+0000 0'0 28:18 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:46:37.845345+0000 0'0 2020-03-11T22:46:37.845345+0000 0 2020-03-11T22:46:49.124 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:35.274435+0000 0'0 27:23 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:46:33.421339+0000 0'0 2020-03-11T22:46:33.421339+0000 0 2020-03-11T22:46:49.124 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:38.122159+0000 0'0 26:22 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:46:35.566806+0000 0'0 2020-03-11T22:46:35.566806+0000 0 2020-03-11T22:46:49.124 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:46:49.124 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:49.125 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:46:49.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:46:49.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:46:49.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:46:49.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:46:49.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:46:49.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:46:49.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:46:49.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:46:49.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:46:49.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:46:49.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:49.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:49.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:46:49.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:46:49.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:46:49.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:46:49.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:46:49.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:46:49.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:46:49.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:46:49.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:46:49.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:46:49.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:46:49.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:46:49.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:49.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:46:49.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:46:49.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:46:49.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:46:49.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:46:49.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:46:49.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:46:49.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:46:49.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:46:49.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:46:49.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:46:49.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:46:49.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:46:49.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:46:49.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:46:49.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:46:49.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:46:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:46:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:46:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:46:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:46:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:46:49.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:46:49.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:46:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' does not exist 2020-03-11T22:46:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:46:50.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:46:50.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:46:50.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:46:50.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:46:50.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:46:50.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:46:51.152 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 1 2020-03-11T22:46:51.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:46:51.753 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:46:51.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:46:52.547 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 1 2020-03-11T22:46:52.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:46:53.230 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:46:53.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:46:53.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:53.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:53.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:53.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:53.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:53.240 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:53.240 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:53.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:53.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:53.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:53.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:53.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:53.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:53.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:53.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:53.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:53.689 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2020-03-11T22:46:53.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:53.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:53.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:53.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:46:53.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:46:53.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:53.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:53.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:46:53.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2020-03-11T22:46:53.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:53.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:46:53.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:53.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:53.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:46:53.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:53.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:46:53.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:46:53.947 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:46:53.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:54.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:46:54.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:54.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:46:54.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:54.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:54.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:46:54.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:54.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:46:54.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:46:54.315 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:46:54.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:54.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672969 2020-03-11T22:46:54.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:54.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:54.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:55.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:46:55.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:55.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:55.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:55.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:55.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:55.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:55.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:55.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:46:55.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:55.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:55.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:55.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:46:55.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:55.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:46:55.955 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records in 2020-03-11T22:46:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records out 2020-03-11T22:46:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0455912 s, 230 MB/s 2020-03-11T22:46:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:46:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:46:55.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:46:55.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:55.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:46:56.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:56.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:46:57.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:46:58.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:58.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:46:59.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:59.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:47:00.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:00.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:47:01.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:01.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:47:02.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:02.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:47:03.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:03.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:47:04.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:04.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:47:05.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:47:05.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:47:05.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:47:05.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:05.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:47:06.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:06.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:47:07.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:47:07.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:47:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:47:07.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:07.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:47:08.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:47:08.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:47:08.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:47:08.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:47:08.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:47:08.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:47:08.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:47:08.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:08.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:47:09.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:09.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:47:10.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:10.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:47:11.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:11.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:47:12.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:12.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:47:13.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:13.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:47:14.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:14.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:47:15.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:15.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:47:16.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:16.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:47:17.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:17.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:47:18.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:18.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:47:19.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:19.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:20.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:47:20.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:20.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:47:20.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:20.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:47:20.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:20.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:47:21.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:47:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:47:21.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:47:21.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:47:21.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:47:21.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:47:21.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:47:21.429 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:47:21.430 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:47:21.430 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:47:21.430 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:47:21.430 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:47:21.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:47:21.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:47:21.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:47:22.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:47:22.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:47:22.466 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:22.466 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:46:51.030376+0000 39'150 39:188 [1] 1 [1] 1 0'0 2020-03-11T22:46:33.421339+0000 0'0 2020-03-11T22:46:33.421339+0000 0 2020-03-11T22:47:22.466 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 34 0 0 0 0 356515840 0 0 102 102 active+clean 2020-03-11T22:46:52.473242+0000 39'102 39:141 [0] 0 [0] 0 0'0 2020-03-11T22:46:35.566806+0000 0'0 2020-03-11T22:46:35.566806+0000 0 2020-03-11T22:47:22.466 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:47:22.466 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:22.467 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:47:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:47:22.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:47:23.161 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 2 2020-03-11T22:47:23.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:47:23.779 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 2 2020-03-11T22:47:23.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:47:28.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:47:28.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:47:28.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:47:28.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:28.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:47:28.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:47:28.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:47:28.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:47:28.893 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 2020-03-11T22:47:28.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:47:28.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:28.893 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:47:28.894 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:47:28.894 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:47:28.894 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:47:28.894 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:47:28.894 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:47:28.894 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:47:28.894 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:47:28.895 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 152, 2020-03-11T22:47:28.895 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:47:28.895 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:47:28.895 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:47:28.895 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:47:28.895 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:47:28.895 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:47:28.896 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:47:28.896 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:47:28.896 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:47:28.896 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:47:28.896 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "2.0", 2020-03-11T22:47:28.896 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 153, 2020-03-11T22:47:28.896 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:47:28.897 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:47:28.897 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:47:28.897 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:47:28.897 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:47:28.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:47:28.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:47:28.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:47:28.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:28.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:28.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:47:28.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.0.asok 2020-03-11T22:47:28.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:47:28.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:47:28.990 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 2020-03-11T22:47:28.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:47:28.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:28.991 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:47:28.991 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:47:28.991 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:47:28.992 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:47:28.992 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:47:28.992 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:47:28.992 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:47:28.992 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "2.0", 2020-03-11T22:47:28.992 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 153, 2020-03-11T22:47:28.992 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:47:28.993 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:47:28.994 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:47:28.994 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:47:28.994 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 152, 2020-03-11T22:47:28.994 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:47:28.994 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:47:28.994 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:47:28.994 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:47:28.995 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:47:28.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:28.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:47:28.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:47:28.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:47:28.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:47:28.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:28.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:47:28.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:47:28.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:47:28.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:28.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:47:29.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:47:29.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:47:29.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:47:29.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:47:29.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:47:29.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:47:29.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:47:29.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:47:29.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:47:29.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:47:29.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:47:29.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:47:29.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:47:29.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:29.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:47:29.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:47:29.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:47:29.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:47:29.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:47:29.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:47:29.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:47:29.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:47:29.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:47:29.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:29.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:29.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:29.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:29.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:29.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:29.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:29.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:29.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:29.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:29.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:29.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:29.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:29.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:29.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:29.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:29.463 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2020-03-11T22:47:29.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:29.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:29.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:29.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836499 2020-03-11T22:47:29.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499' 2020-03-11T22:47:29.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:29.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:29.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672978 2020-03-11T22:47:29.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2020-03-11T22:47:29.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:29.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836499 2020-03-11T22:47:29.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:29.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:29.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836499 2020-03-11T22:47:29.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:29.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836499 2020-03-11T22:47:29.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-03-11T22:47:29.724 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836499 2020-03-11T22:47:29.724 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:30.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836499 2020-03-11T22:47:30.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:31.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:31.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:31.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836499 2020-03-11T22:47:31.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:31.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672978 2020-03-11T22:47:31.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:31.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:31.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672978 2020-03-11T22:47:31.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:31.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672978 2020-03-11T22:47:31.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-03-11T22:47:31.678 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672978 2020-03-11T22:47:31.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:32.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 42949672978 2020-03-11T22:47:32.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:32.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:32.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:32.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:47:32.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:32.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:32.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:32.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:32.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:32.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:32.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:33.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:33.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:33.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:33.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:33.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:33.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:47:33.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:33.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:47:33.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:33.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:33.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:33.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:33.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:33.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:33.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:33.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:33.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:33.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:34.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:34.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:34.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:34.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:34.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:34.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:34.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:34.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:34.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:34.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:34.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:34.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:34.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:35.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30402179 2020-03-11T22:47:35.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30402179 '!=' null 2020-03-11T22:47:35.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:35.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:35.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:35.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:35.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:35.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:35.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:35.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:35.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:35.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:35.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:35.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:35.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:35.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:35.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:36.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:36.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:36.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:36.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:36.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25735998 2020-03-11T22:47:36.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25735998 '!=' null 2020-03-11T22:47:36.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:36.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:36.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:36.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:36.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:36.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:36.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:36.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:36.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:36.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:36.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:37.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:37.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:37.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:37.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:37.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:37.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:37.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:37.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:37.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:37.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:37.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:37.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:37.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:38.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25735998 2020-03-11T22:47:38.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25735998 '!=' null 2020-03-11T22:47:38.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:38.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:38.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:38.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:38.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:38.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:38.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:38.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:38.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:38.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:38.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:38.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:38.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:38.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:39.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:39.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:39.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:39.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:39.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:39.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:39.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:39.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:39.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:39.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198232 2020-03-11T22:47:39.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198232 '!=' null 2020-03-11T22:47:39.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:39.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:39.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:39.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:39.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:39.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:39.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:39.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:39.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:39.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:39.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:39.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:39.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:39.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:39.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:40.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:40.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:40.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:40.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:41.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=36692872 2020-03-11T22:47:41.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 36692872 '!=' null 2020-03-11T22:47:41.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:41.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:41.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:41.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:41.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:41.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:41.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:41.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:41.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:41.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:41.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:41.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:41.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:42.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:42.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:42.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:42.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:42.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:42.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:42.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:42.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:42.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:42.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25335782 2020-03-11T22:47:42.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25335782 '!=' null 2020-03-11T22:47:42.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:42.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:42.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:42.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:42.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:42.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:42.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:42.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:42.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:42.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:42.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:43.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:43.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:43.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:43.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:43.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:43.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:43.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:43.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:43.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:43.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:43.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:43.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:43.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:44.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25335782 2020-03-11T22:47:44.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25335782 '!=' null 2020-03-11T22:47:44.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:44.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:44.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:44.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:44.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:44.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:44.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:44.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:44.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:44.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:44.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:44.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:44.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:44.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:44.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:45.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:45.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:45.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:45.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:45.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:45.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:45.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:45.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:45.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:45.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39314494 2020-03-11T22:47:45.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39314494 '!=' null 2020-03-11T22:47:45.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:45.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:45.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:45.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:45.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:45.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:45.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:45.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:45.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:45.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:45.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:46.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:46.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:46.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:46.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:46.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:46.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:46.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:46.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:46.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:46.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:46.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:46.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:46.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:47.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704336 2020-03-11T22:47:47.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704336 '!=' null 2020-03-11T22:47:47.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:47.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:47.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:47.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:47.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:47.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:47.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:47.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:47.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:47.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:47.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:47.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:47.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:47.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:47.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:48.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:48.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:48.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:48.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:48.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37567383 2020-03-11T22:47:48.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37567383 '!=' null 2020-03-11T22:47:48.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:48.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:48.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:48.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:48.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:48.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:48.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:48.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:48.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:48.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:48.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:49.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:49.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:49.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:49.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:49.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:49.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:49.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:49.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:49.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:49.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:49.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:49.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:49.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:50.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37567513 2020-03-11T22:47:50.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37567513 '!=' null 2020-03-11T22:47:50.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:50.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:50.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:50.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:50.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:50.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:50.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:50.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:50.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:50.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:50.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:50.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:50.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:51.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:51.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:51.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:51.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:51.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:51.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:51.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:51.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:51.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:51.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37567513 2020-03-11T22:47:51.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37567513 '!=' null 2020-03-11T22:47:51.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:51.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:51.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:51.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:51.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:51.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:51.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:51.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:51.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:51.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:51.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:52.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:52.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:52.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:52.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:52.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:52.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:52.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:52.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:52.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:52.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:52.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:52.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:52.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704970 2020-03-11T22:47:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704970 '!=' null 2020-03-11T22:47:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:53.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:53.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:53.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:53.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:53.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:53.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:53.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:53.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:53.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:53.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:53.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:53.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:54.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:54.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:54.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:54.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:54.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:54.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:54.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:54.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:54.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:54.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=43683687 2020-03-11T22:47:54.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 43683687 '!=' null 2020-03-11T22:47:54.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:54.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:54.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:54.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:54.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:54.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:54.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:54.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:54.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:55.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:55.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:55.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:55.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:55.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:55.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:55.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:55.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:55.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:55.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:55.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:55.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:55.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:56.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=43683687 2020-03-11T22:47:56.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 43683687 '!=' null 2020-03-11T22:47:56.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:56.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:56.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:56.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:56.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:56.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:56.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:56.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:56.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:56.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:56.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:56.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:56.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:56.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:56.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:56.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:56.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:56.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:57.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41936998 2020-03-11T22:47:57.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41936998 '!=' null 2020-03-11T22:47:57.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:57.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:57.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:57.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:57.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:57.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:57.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:57.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:57.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:57.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:57.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:57.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:57.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:57.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:57.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:58.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:47:58.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:58.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:58.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:58.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:58.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:58.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:58.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:58.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:58.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26036394 2020-03-11T22:47:58.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26036394 '!=' null 2020-03-11T22:47:58.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:47:58.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:58.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:58.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:58.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:58.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:58.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:58.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:58.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:58.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:58.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:59.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:47:59.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:59.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:59.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:59.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:47:59.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:59.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:59.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:59.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:59.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:59.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:59.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:59.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:59.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:00.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:48:00.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:00.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:00.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:00.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:48:00.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:48:00.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:00.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:00.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:00.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:00.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:00.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:00.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:00.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=5056574 2020-03-11T22:48:00.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 5056574 '!=' null 2020-03-11T22:48:00.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:00.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:00.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:01.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:01.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:01.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:01.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:01.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:01.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:01.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:01.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:01.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:48:01.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:01.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:01.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:01.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:48:01.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:48:01.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:48:01.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:48:01.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:48:01.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:48:01.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:48:01.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:48:01.845 INFO:tasks.workunit.client.0.smithi130.stdout:TEST PASSED 2020-03-11T22:48:02.335 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' does not exist 2020-03-11T22:48:02.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:48:02.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:48:02.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:48:02.945 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' does not exist 2020-03-11T22:48:02.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:48:02.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:02.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:02.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:02.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:02.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:03.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:03.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:48:03.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:48:03.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:48:03.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:03.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:48:03.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:03.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:03.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:03.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:03.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:03.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:03.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:03.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:03.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:03.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:03.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:03.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:03.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:03.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:03.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:03.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:03.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:03.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:03.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:48:03.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:03.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:03.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:03.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56067 2020-03-11T22:48:03.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:03.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:03.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:48:03.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:48:03.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:03.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:03.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:03.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:03.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:03.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:03.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:03.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:03.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:03.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:03.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:03.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:03.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:03.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:03.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:03.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:03.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:48:03.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:03.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:03.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:03.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56067 2020-03-11T22:48:03.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:03.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:03.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:48:03.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:48:03.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:03.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:03.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56067 2020-03-11T22:48:03.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:48:03.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:48:03.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:48:03.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:48:03.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:48:03.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:48:03.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:48:03.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:48:03.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:48:03.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:48:03.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:48:03.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:48:03.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:48:03.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:48:03.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:48:03.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:48:03.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:48:03.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:48:03.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:48:03.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:03.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:03.634 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:03.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:03.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:03.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:03.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:48:03.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:48:03.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:48:03.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:03.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:03.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:48:03.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:48:03.802 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:03.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:03.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:03.803 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:03.803 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:03.803 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:03.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-mon.a.asok 2020-03-11T22:48:03.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:03.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56067/ceph-mon.a.asok config get fsid 2020-03-11T22:48:03.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:48:03.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:03.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:03.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:48:03.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:48:03.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:03.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:03.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:03.890 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:03.890 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:03.891 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:03.891 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-mon.a.asok 2020-03-11T22:48:03.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:03.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56067/ceph-mon.a.asok config get mon_host 2020-03-11T22:48:03.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:48:03.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:48:03.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:48:03.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:48:03.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:48:03.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:48:03.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:48:04.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:48:04.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:04.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:04.859 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:04.859 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:04.859 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:04.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:04.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:04.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:04.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:48:04.882 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:48:04.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:48:04.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:04.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:48:04.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:48:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:48:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:48:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:48:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:04.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:04.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:04.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:04.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:04.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:04.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:04.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:04.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:04.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:04.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:04.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:04.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:04.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:04.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:04.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:04.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:04.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:04.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:04.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:48:04.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:04.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=018b24df-2846-4c02-9164-cee7e92b4b79 2020-03-11T22:48:04.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 018b24df-2846-4c02-9164-cee7e92b4b79' 2020-03-11T22:48:04.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:04.895 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 018b24df-2846-4c02-9164-cee7e92b4b79 2020-03-11T22:48:04.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCkamle8EvYNRAA6SM4jwmlS20C/5oVvpyU3A== 2020-03-11T22:48:04.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCkamle8EvYNRAA6SM4jwmlS20C/5oVvpyU3A=="}' 2020-03-11T22:48:04.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 018b24df-2846-4c02-9164-cee7e92b4b79 -i td/osd-backfill-prio/0/new.json 2020-03-11T22:48:05.240 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:48:05.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:48:05.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCkamle8EvYNRAA6SM4jwmlS20C/5oVvpyU3A== --osd-uuid 018b24df-2846-4c02-9164-cee7e92b4b79 2020-03-11T22:48:05.277 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:05.273+0000 7f6d035e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:05.278 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:05.275+0000 7f6d035e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:05.280 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:05.276+0000 7f6d035e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:05.280 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:05.277+0000 7f6d035e4ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:05.281 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:05.277+0000 7f6d035e4ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:05.281 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:05.277+0000 7f6d035e4ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:48:05.316 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:05.313+0000 7f6d035e4ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:48:08.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:48:08.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:08.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:48:08.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:08.302 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:48:08.756 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:48:08.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:48:08.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:08.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:08.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:08.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:08.779 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:08.776+0000 7f2cf8d4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:08.781 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:08.778+0000 7f2cf8d4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:08.783 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:08.779+0000 7f2cf8d4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:09.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:48:09.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:09.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:48:09.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:09.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:09.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:09.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:09.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:09.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:09.107 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:48:09.306 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:09.303+0000 7f2cf8d4cec0 -1 Falling back to public interface 2020-03-11T22:48:09.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:10.049 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:10.045+0000 7f2cf8d4cec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:48:10.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:10.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:10.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:10.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:10.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:10.454 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:48:10.811 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/63051,v1:127.0.0.1:6803/63051] [v2:127.0.0.1:6804/63051,v1:127.0.0.1:6805/63051] exists,up 018b24df-2846-4c02-9164-cee7e92b4b79 2020-03-11T22:48:10.811 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 2184a939-c503-4a31-b172-0679728edbf2 2020-03-11T22:48:10.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:10.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:10.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:10.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:10.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:48:10.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:10.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:10.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:48:10.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:10.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:48:10.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:10.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:10.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:10.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:10.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:48:10.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:48:10.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:10.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:10.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:10.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:10.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:10.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:10.816 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:10.816 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:10.817 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:10.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:10.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:10.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:10.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:10.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:10.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:10.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:10.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:10.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:10.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:10.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:10.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:10.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:48:10.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:10.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2184a939-c503-4a31-b172-0679728edbf2 2020-03-11T22:48:10.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 2184a939-c503-4a31-b172-0679728edbf2' 2020-03-11T22:48:10.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:10.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCqamleZ3ghMRAAW4lfBOO4aoyspFsFYtNxAg== 2020-03-11T22:48:10.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCqamleZ3ghMRAAW4lfBOO4aoyspFsFYtNxAg=="}' 2020-03-11T22:48:10.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2184a939-c503-4a31-b172-0679728edbf2 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:48:11.346 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:48:11.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:48:11.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCqamleZ3ghMRAAW4lfBOO4aoyspFsFYtNxAg== --osd-uuid 2184a939-c503-4a31-b172-0679728edbf2 2020-03-11T22:48:11.382 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:11.378+0000 7fd849551ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:11.389 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:11.386+0000 7fd849551ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:11.390 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:11.387+0000 7fd849551ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:11.391 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:11.388+0000 7fd849551ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:11.391 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:11.388+0000 7fd849551ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:11.391 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:11.388+0000 7fd849551ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:48:11.514 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:11.511+0000 7fd849551ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:48:14.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:48:14.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:14.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:48:14.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:14.566 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:48:15.021 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:48:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:48:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:15.045 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:15.042+0000 7f73ca263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:15.047 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:15.044+0000 7f73ca263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:15.049 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:15.046+0000 7f73ca263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:15.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:48:15.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:15.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:48:15.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:15.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:15.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:15.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:15.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:15.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:15.377 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:48:15.569 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:15.566+0000 7f73ca263ec0 -1 Falling back to public interface 2020-03-11T22:48:15.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:16.369+0000 7f73ca263ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:48:16.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:16.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:16.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:16.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:16.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:16.727 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:48:17.107 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63406,v1:127.0.0.1:6811/63406] [v2:127.0.0.1:6812/63406,v1:127.0.0.1:6813/63406] exists,up 2184a939-c503-4a31-b172-0679728edbf2 2020-03-11T22:48:17.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:17.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:17.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:17.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:17.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:48:17.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:17.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:17.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:48:17.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:17.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:48:17.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:17.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:17.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:17.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:17.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:17.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:48:17.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:17.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=83a10819-002d-4f91-8cf3-bd988e3edeff 2020-03-11T22:48:17.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 83a10819-002d-4f91-8cf3-bd988e3edeff' 2020-03-11T22:48:17.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:17.113 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 83a10819-002d-4f91-8cf3-bd988e3edeff 2020-03-11T22:48:17.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCxamleYY89BxAACwR632S5lSFSg1yEyjsUDw== 2020-03-11T22:48:17.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCxamleYY89BxAACwR632S5lSFSg1yEyjsUDw=="}' 2020-03-11T22:48:17.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 83a10819-002d-4f91-8cf3-bd988e3edeff -i td/osd-backfill-prio/2/new.json 2020-03-11T22:48:17.620 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:48:17.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:48:17.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCxamleYY89BxAACwR632S5lSFSg1yEyjsUDw== --osd-uuid 83a10819-002d-4f91-8cf3-bd988e3edeff 2020-03-11T22:48:17.657 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:17.654+0000 7fc754b78ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:17.659 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:17.655+0000 7fc754b78ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:17.660 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:17.657+0000 7fc754b78ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:17.661 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:17.657+0000 7fc754b78ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:17.661 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:17.657+0000 7fc754b78ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:17.661 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:17.657+0000 7fc754b78ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:48:17.720 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:17.717+0000 7fc754b78ec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:48:20.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:48:20.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:20.707 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:48:20.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:48:20.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:21.166 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:48:21.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:48:21.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:21.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:21.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:21.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:21.189 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:21.186+0000 7fd5e4f27ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:21.190 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:21.187+0000 7fd5e4f27ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:21.192 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:21.189+0000 7fd5e4f27ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:48:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:48:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:21.516 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:48:21.707 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:21.703+0000 7fd5e4f27ec0 -1 Falling back to public interface 2020-03-11T22:48:21.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:22.496 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:22.492+0000 7fd5e4f27ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:48:22.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:22.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:22.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:22.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:22.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:22.868 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:48:23.233 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63761,v1:127.0.0.1:6819/63761] [v2:127.0.0.1:6820/63761,v1:127.0.0.1:6821/63761] exists,up 83a10819-002d-4f91-8cf3-bd988e3edeff 2020-03-11T22:48:23.233 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 da9221de-4e97-4e45-876d-16ed9f2f444a 2020-03-11T22:48:23.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:48:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:48:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:48:23.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:23.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:23.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:23.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:23.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:23.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:23.236 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:23.237 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:23.237 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:23.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:23.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:23.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:23.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:23.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:48:23.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:23.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=da9221de-4e97-4e45-876d-16ed9f2f444a 2020-03-11T22:48:23.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 da9221de-4e97-4e45-876d-16ed9f2f444a' 2020-03-11T22:48:23.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC3amle/mKtDhAANa05uu3ITyMt6GQcvQKl8g== 2020-03-11T22:48:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC3amle/mKtDhAANa05uu3ITyMt6GQcvQKl8g=="}' 2020-03-11T22:48:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new da9221de-4e97-4e45-876d-16ed9f2f444a -i td/osd-backfill-prio/3/new.json 2020-03-11T22:48:23.779 INFO:tasks.workunit.client.0.smithi130.stdout:3 2020-03-11T22:48:23.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:48:23.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC3amle/mKtDhAANa05uu3ITyMt6GQcvQKl8g== --osd-uuid da9221de-4e97-4e45-876d-16ed9f2f444a 2020-03-11T22:48:23.815 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:23.811+0000 7f8742c04ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:23.836 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:23.832+0000 7f8742c04ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:23.838 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:23.834+0000 7f8742c04ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:23.838 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:23.835+0000 7f8742c04ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:23.838 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:23.835+0000 7f8742c04ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:23.839 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:23.835+0000 7f8742c04ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:48:23.910 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:23.906+0000 7f8742c04ec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:48:26.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:48:26.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:26.972 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2020-03-11T22:48:26.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:48:26.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:27.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:48:27.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:27.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:27.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:27.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:27.431 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2020-03-11T22:48:27.453 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:27.450+0000 7f09ee773ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:27.455 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:27.452+0000 7f09ee773ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:27.456 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:27.453+0000 7f09ee773ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:27.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:48:27.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:27.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:48:27.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:27.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:27.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:27.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:27.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:27.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:48:27.779 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:48:27.972 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:27.968+0000 7f09ee773ec0 -1 Falling back to public interface 2020-03-11T22:48:28.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:28.745 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:28.742+0000 7f09ee773ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:48:29.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:29.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:48:29.131 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:48:29.501 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/64116,v1:127.0.0.1:6827/64116] [v2:127.0.0.1:6828/64116,v1:127.0.0.1:6829/64116] exists,up da9221de-4e97-4e45-876d-16ed9f2f444a 2020-03-11T22:48:29.501 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 48f9b13e-3c5f-473a-a836-57a758c5bd05 2020-03-11T22:48:29.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:29.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:48:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:48:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:48:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:48:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:29.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:48:29.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:48:29.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:29.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:29.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:29.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:48:29.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:48:29.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:29.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:29.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:48:29.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:29.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:29.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:29.504 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:48:29.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' 2020-03-11T22:48:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:29.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:29.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:48:29.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:48:29.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:29.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:29.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:29.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:29.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:29.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:48:29.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:29.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=48f9b13e-3c5f-473a-a836-57a758c5bd05 2020-03-11T22:48:29.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 48f9b13e-3c5f-473a-a836-57a758c5bd05' 2020-03-11T22:48:29.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:29.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC9amleGN6oHhAAqpayeR26z9qaQUJDJ6JxiQ== 2020-03-11T22:48:29.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC9amleGN6oHhAAqpayeR26z9qaQUJDJ6JxiQ=="}' 2020-03-11T22:48:29.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 48f9b13e-3c5f-473a-a836-57a758c5bd05 -i td/osd-backfill-prio/4/new.json 2020-03-11T22:48:30.078 INFO:tasks.workunit.client.0.smithi130.stdout:4 2020-03-11T22:48:30.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:48:30.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC9amleGN6oHhAAqpayeR26z9qaQUJDJ6JxiQ== --osd-uuid 48f9b13e-3c5f-473a-a836-57a758c5bd05 2020-03-11T22:48:30.113 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:30.110+0000 7ff61a67cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:30.115 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:30.112+0000 7ff61a67cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:30.116 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:30.112+0000 7ff61a67cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:30.116 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:30.113+0000 7ff61a67cec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:30.116 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:30.113+0000 7ff61a67cec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:30.117 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:30.113+0000 7ff61a67cec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:48:30.204 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:30.201+0000 7ff61a67cec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:48:33.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:48:33.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:33.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:48:33.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:33.364 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2020-03-11T22:48:33.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:48:33.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=d8b0751a-ed0f-4369-94af-aeb86d3dfde2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56067/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:33.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:33.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:33.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:33.820 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2020-03-11T22:48:33.843 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:33.841+0000 7fa1ecc1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.845 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:33.843+0000 7fa1ecc1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:33.846 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:33.843+0000 7fa1ecc1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:34.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:48:34.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:34.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:48:34.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:34.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:34.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:34.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:34.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:34.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:48:34.185 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:48:34.363 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:34.360+0000 7fa1ecc1fec0 -1 Falling back to public interface 2020-03-11T22:48:34.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:35.128 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:48:35.125+0000 7fa1ecc1fec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:48:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:48:35.552 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:48:35.918 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64471,v1:127.0.0.1:6835/64471] [v2:127.0.0.1:6836/64471,v1:127.0.0.1:6837/64471] exists,up 48f9b13e-3c5f-473a-a836-57a758c5bd05 2020-03-11T22:48:35.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:35.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:35.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:35.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:48:35.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:35.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:48:35.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:48:36.618 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' already exists 2020-03-11T22:48:36.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:37.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:48:38.266 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 2 2020-03-11T22:48:38.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:38.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:48:38.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:48:39.019 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' already exists 2020-03-11T22:48:39.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:40.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:48:40.722 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 2 2020-03-11T22:48:40.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:40.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:48:40.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:48:41.186 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' already exists 2020-03-11T22:48:41.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:48:43.069 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 2 2020-03-11T22:48:43.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:43.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:48:43.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:48:43.583 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test4' already exists 2020-03-11T22:48:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:44.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:48:45.426 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 4 size to 2 2020-03-11T22:48:45.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:45.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:48:45.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:48:45.950 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test5' already exists 2020-03-11T22:48:45.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:46.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:48:47.563 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 5 size to 2 2020-03-11T22:48:47.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:47.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:48:47.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:48:48.036 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test6' already exists 2020-03-11T22:48:48.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:49.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:48:49.660 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 6 size to 2 2020-03-11T22:48:49.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:49.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:48:49.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:48:50.389 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test7' already exists 2020-03-11T22:48:50.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:51.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:48:52.043 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 7 size to 2 2020-03-11T22:48:52.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:52.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:48:52.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:48:52.576 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test8' already exists 2020-03-11T22:48:52.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:53.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:48:54.215 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 8 size to 2 2020-03-11T22:48:54.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:54.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:48:54.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:48:54.672 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test9' already exists 2020-03-11T22:48:54.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:55.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:48:56.497 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 9 size to 2 2020-03-11T22:48:56.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:56.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:48:56.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:48:56.997 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test10' already exists 2020-03-11T22:48:57.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:48:58.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:48:58.870 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 10 size to 2 2020-03-11T22:48:58.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:49:03.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:49:03.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:03.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:03.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:03.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:03.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:03.882 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:03.882 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:03.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:03.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:03.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:03.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:03.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:03.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:03.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:03.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:04.330 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:49:04.330 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:49:04.330 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:49:04.330 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:49:04.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:04.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:04.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:04.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:04.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672973 2020-03-11T22:49:04.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973' 2020-03-11T22:49:04.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:04.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:04.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:49:04.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509451' 2020-03-11T22:49:04.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:04.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:04.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345929 2020-03-11T22:49:04.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509451 3-85899345929' 2020-03-11T22:49:04.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:04.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:04.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182408 2020-03-11T22:49:04.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509451 3-85899345929 4-107374182408' 2020-03-11T22:49:04.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:04.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:49:04.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:04.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:04.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:49:04.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:04.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:49:04.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:49:04.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:04.963 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:49:05.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:49:05.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:06.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:06.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:06.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:49:06.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:06.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672973 2020-03-11T22:49:06.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:06.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:06.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672973 2020-03-11T22:49:06.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:06.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672973 2020-03-11T22:49:06.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2020-03-11T22:49:06.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:06.671 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672973 2020-03-11T22:49:07.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672973 2020-03-11T22:49:07.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:07.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:49:07.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:07.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:07.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:49:07.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:07.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:49:07.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:49:07.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:07.020 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:49:07.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:49:07.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:07.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345929 2020-03-11T22:49:07.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:07.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:07.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345929 2020-03-11T22:49:07.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:07.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345929 2020-03-11T22:49:07.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2020-03-11T22:49:07.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:07.367 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345929 2020-03-11T22:49:07.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345930 -lt 85899345929 2020-03-11T22:49:07.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:07.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182408 2020-03-11T22:49:07.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:07.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:07.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182408 2020-03-11T22:49:07.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:07.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182408 2020-03-11T22:49:07.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2020-03-11T22:49:07.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:07.717 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182408 2020-03-11T22:49:08.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182408 -lt 107374182408 2020-03-11T22:49:08.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:08.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:08.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:08.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:49:08.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:08.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:08.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:08.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:08.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:08.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:08.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:08.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:49:08.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:08.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:08.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:09.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:49:09.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:09.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:09.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:49:09.644 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:09.644 INFO:tasks.workunit.client.0.smithi130.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:59.733955+0000 0'0 82:20 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:48:56.812119+0000 0'0 2020-03-11T22:48:56.812119+0000 0 2020-03-11T22:49:09.645 INFO:tasks.workunit.client.0.smithi130.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:56.655675+0000 0'0 80:25 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:54.519600+0000 0'0 2020-03-11T22:48:54.519600+0000 0 2020-03-11T22:49:09.645 INFO:tasks.workunit.client.0.smithi130.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:54.805492+0000 0'0 81:28 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:48:52.385175+0000 0'0 2020-03-11T22:48:52.385175+0000 0 2020-03-11T22:49:09.645 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:52.360559+0000 0'0 80:34 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:48:50.039875+0000 0'0 2020-03-11T22:48:50.039875+0000 0 2020-03-11T22:49:09.645 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:41.428927+0000 0'0 80:50 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:48:38.645600+0000 0'0 2020-03-11T22:48:38.645600+0000 0 2020-03-11T22:49:09.645 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:38.864286+0000 0'0 80:46 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:48:36.415824+0000 0'0 2020-03-11T22:48:36.415824+0000 0 2020-03-11T22:49:09.645 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:43.613705+0000 0'0 80:45 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:41.034094+0000 0'0 2020-03-11T22:48:41.034094+0000 0 2020-03-11T22:49:09.646 INFO:tasks.workunit.client.0.smithi130.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:46.223031+0000 0'0 81:47 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:48:43.382835+0000 0'0 2020-03-11T22:48:43.382835+0000 0 2020-03-11T22:49:09.646 INFO:tasks.workunit.client.0.smithi130.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:47.818430+0000 0'0 80:31 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:48:45.762712+0000 0'0 2020-03-11T22:48:45.762712+0000 0 2020-03-11T22:49:09.646 INFO:tasks.workunit.client.0.smithi130.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:48:50.701824+0000 0'0 81:38 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:48:47.862758+0000 0'0 2020-03-11T22:48:47.862758+0000 0 2020-03-11T22:49:09.646 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:49:09.646 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:09.647 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:49:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:49:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:49:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:49:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:49:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:49:09.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:49:09.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:49:09.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:49:09.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:49:09.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:49:09.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:49:09.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:49:09.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:09.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:09.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:49:09.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:09.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:09.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:09.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:49:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:49:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:49:09.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:49:09.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:49:09.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:49:09.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:49:09.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:09.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:49:09.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:10.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:49:10.335 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:49:10.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:10.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:49:10.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:49:10.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:10.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:49:10.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:10.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:10.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:10.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:49:10.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:10.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:49:10.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:49:10.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:49:10.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:49:10.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:49:10.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:10.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:49:10.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:11.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:49:11.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:49:11.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:11.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:49:11.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:49:11.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:11.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:49:11.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:11.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:49:11.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:49:11.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:11.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:49:11.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:49:11.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:11.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:49:11.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:11.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:49:11.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:11.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:49:11.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:11.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:49:11.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:49:11.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:11.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:49:11.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:49:12.054 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:12.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:49:12.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:12.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:49:12.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:49:12.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:49:12.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:49:12.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:49:12.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:49:12.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:49:12.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:49:12.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:49:12.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:49:12.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:12.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:12.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:49:12.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:12.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:49:12.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:49:12.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:49:12.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:49:12.554 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' does not exist 2020-03-11T22:49:12.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:12.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:49:12.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:12.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:49:12.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:49:12.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:49:12.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:49:13.101 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test4' does not exist 2020-03-11T22:49:13.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:13.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:49:13.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:49:13.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:49:13.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:49:13.674 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test5' does not exist 2020-03-11T22:49:13.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:13.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:49:13.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:49:13.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:49:13.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:49:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test6' does not exist 2020-03-11T22:49:14.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:49:14.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:49:14.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:49:14.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:49:14.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:49:14.697 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test8' does not exist 2020-03-11T22:49:14.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:14.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:49:14.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:49:14.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:49:14.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:49:15.133 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test9' does not exist 2020-03-11T22:49:15.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:15.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:49:15.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:49:15.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:49:15.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:49:15.600 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test10' does not exist 2020-03-11T22:49:15.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:49:16.315 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 1 2020-03-11T22:49:16.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:49:17.317 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 7 size to 1 2020-03-11T22:49:17.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:49:17.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:17.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:17.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:17.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:17.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:17.327 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:17.327 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:17.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:17.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:17.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:17.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:17.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:17.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:17.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:17.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:17.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:49:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:49:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:49:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:49:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:17.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:17.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836498 2020-03-11T22:49:17.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498' 2020-03-11T22:49:17.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:17.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:18.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672978 2020-03-11T22:49:18.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978' 2020-03-11T22:49:18.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:18.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509455 2020-03-11T22:49:18.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509455' 2020-03-11T22:49:18.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:18.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345934 2020-03-11T22:49:18.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509455 3-85899345934' 2020-03-11T22:49:18.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:18.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182413 2020-03-11T22:49:18.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509455 3-85899345934 4-107374182413' 2020-03-11T22:49:18.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:18.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836498 2020-03-11T22:49:18.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:18.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:18.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836498 2020-03-11T22:49:18.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:18.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836498 2020-03-11T22:49:18.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-03-11T22:49:18.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:18.418 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836498 2020-03-11T22:49:18.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836498 -lt 21474836498 2020-03-11T22:49:18.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:18.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672978 2020-03-11T22:49:18.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:18.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:18.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672978 2020-03-11T22:49:18.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:18.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672978 2020-03-11T22:49:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-03-11T22:49:18.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:18.774 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672978 2020-03-11T22:49:19.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 42949672978 2020-03-11T22:49:19.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:19.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509455 2020-03-11T22:49:19.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:19.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:19.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509455 2020-03-11T22:49:19.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:19.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509455 2020-03-11T22:49:19.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2020-03-11T22:49:19.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:19.123 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509455 2020-03-11T22:49:19.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509453 -lt 64424509455 2020-03-11T22:49:19.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:20.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:20.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:20.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509456 -lt 64424509455 2020-03-11T22:49:20.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:20.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345934 2020-03-11T22:49:20.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:20.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:20.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345934 2020-03-11T22:49:20.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:20.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345934 2020-03-11T22:49:20.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2020-03-11T22:49:20.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:20.842 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345934 2020-03-11T22:49:21.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345935 -lt 85899345934 2020-03-11T22:49:21.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:21.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182413 2020-03-11T22:49:21.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:21.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182413 2020-03-11T22:49:21.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:21.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182413 2020-03-11T22:49:21.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2020-03-11T22:49:21.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:21.187 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182413 2020-03-11T22:49:21.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182413 -lt 107374182413 2020-03-11T22:49:21.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:21.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:21.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:21.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:49:21.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:21.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:21.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:21.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:21.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:21.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:21.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:22.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:49:22.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:22.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:22.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:22.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:49:22.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:22.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:22.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:49:22.813 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records in 2020-03-11T22:49:22.813 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records out 2020-03-11T22:49:22.813 INFO:tasks.workunit.client.0.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0457183 s, 229 MB/s 2020-03-11T22:49:22.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:49:22.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:49:22.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:49:22.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:22.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:49:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:49:23.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:23.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:49:24.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:24.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:49:24.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:24.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:49:24.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:24.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:49:25.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:25.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:49:25.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:25.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:49:26.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:49:26.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:26.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:49:27.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:27.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:49:27.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:27.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:49:28.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:28.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:49:28.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:28.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:49:28.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:28.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:49:29.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:29.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:49:29.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:29.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:49:30.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:30.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:49:30.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:30.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:49:31.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:31.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:49:31.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:31.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:49:32.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:32.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:49:32.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:32.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:49:33.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:33.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:49:33.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:33.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:49:33.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:33.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:49:34.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:34.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:49:34.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:34.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:49:35.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:35.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:49:35.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:35.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:49:36.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:36.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:49:36.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:36.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:49:36.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:36.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:49:37.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:37.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:49:37.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:37.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:49:38.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:38.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:49:38.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:38.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:49:39.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:39.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:49:39.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:39.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:49:40.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:40.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:49:40.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:40.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:49:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:41.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:49:41.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:41.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:49:42.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:42.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:49:42.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:42.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:49:42.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:42.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:49:43.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:43.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:49:43.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:43.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:49:44.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:44.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:49:44.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:44.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:49:45.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:49:45.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:49:45.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:49:45.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:49:45.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:49:45.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:49:45.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:49:45.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:45.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:49:46.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:46.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:49:46.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:46.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:49:46.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:46.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:49:46.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:46.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:49:47.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:49:47.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:49:47.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:49:47.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:47.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:49:48.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:49:48.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:49:48.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:49:48.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:48.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:49:49.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:49:49.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:49:49.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:49:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:49:50.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:50.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:49:51.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:49:51.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:49:51.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:49:51.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:51.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:49:52.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:49:52.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:49:52.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:49:52.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:52.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:49:53.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:53.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:49:54.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:54.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:49:55.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:55.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:49:56.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:56.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:49:57.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:49:57.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:49:57.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:49:57.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:49:57.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:57.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:49:58.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:49:58.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:49:58.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:49:58.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:58.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:49:59.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:49:59.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:49:59.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:49:59.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:59.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:50:00.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:50:00.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:50:00.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:50:00.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:00.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:50:01.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:50:01.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:50:01.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:50:01.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:01.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:50:02.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:50:02.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:50:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:50:02.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:02.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:50:03.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:50:03.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:50:03.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:50:03.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:50:04.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:50:05.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:50:05.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:50:05.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:50:05.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:50:06.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:50:06.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:50:06.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:50:06.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:50:07.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:50:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:50:09.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:50:10.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:50:10.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:50:10.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:50:10.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:50:10.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:50:10.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:50:10.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:50:10.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:50:10.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:10.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:50:10.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:50:10.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:10.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:50:11.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:50:11.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:11.631 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:11.632 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 39 0 0 0 0 408944640 0 0 17 17 active+clean 2020-03-11T22:49:17.070064+0000 94'117 94:164 [1] 1 [1] 1 0'0 2020-03-11T22:48:50.039875+0000 0'0 2020-03-11T22:48:50.039875+0000 0 2020-03-11T22:50:11.632 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:48:38.864286+0000 94'150 94:202 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:48:36.415824+0000 0'0 2020-03-11T22:48:36.415824+0000 0 2020-03-11T22:50:11.632 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:17.119773+0000 94'150 94:208 [1] 1 [1] 1 0'0 2020-03-11T22:48:41.034094+0000 0'0 2020-03-11T22:48:41.034094+0000 0 2020-03-11T22:50:11.632 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:50:11.632 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:11.633 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:50:11.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:50:11.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:50:12.317 INFO:tasks.workunit.client.0.smithi130.stderr:nobackfill is set 2020-03-11T22:50:12.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:50:13.047 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2020-03-11T22:50:13.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:50:13.916 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 7 size to 2 2020-03-11T22:50:13.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:50:15.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:15.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:15.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:15.928 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:15.928 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:15.928 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:15.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:15.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:15.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:16.025 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:16.026 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:16.026 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:16.026 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:16.026 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:16.026 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:16.026 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:16.026 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:16.027 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:50:16.027 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:16.027 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:16.027 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:16.027 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:16.027 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:16.027 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:16.028 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:16.028 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:16.028 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:16.028 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:16.028 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:16.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:16.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:16.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:16.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:16.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:50:16.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:50:18.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:18.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:18.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:18.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:50:18.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:18.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:18.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:50:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:50:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:50:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:50:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:19.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:19.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836512 2020-03-11T22:50:19.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512' 2020-03-11T22:50:19.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:19.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:19.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672992 2020-03-11T22:50:19.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992' 2020-03-11T22:50:19.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:19.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:19.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509470 2020-03-11T22:50:19.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470' 2020-03-11T22:50:19.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:19.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345948 2020-03-11T22:50:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-85899345948' 2020-03-11T22:50:19.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:19.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182427 2020-03-11T22:50:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-85899345948 4-107374182427' 2020-03-11T22:50:19.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:19.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836512 2020-03-11T22:50:19.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:19.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:19.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836512 2020-03-11T22:50:19.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:19.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836512 2020-03-11T22:50:19.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2020-03-11T22:50:19.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:19.719 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836512 2020-03-11T22:50:20.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836510 -lt 21474836512 2020-03-11T22:50:20.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:21.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:21.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:21.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836512 -lt 21474836512 2020-03-11T22:50:21.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:21.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672992 2020-03-11T22:50:21.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:21.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:21.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672992 2020-03-11T22:50:21.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:21.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672992 2020-03-11T22:50:21.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2020-03-11T22:50:21.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:21.423 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672992 2020-03-11T22:50:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672992 -lt 42949672992 2020-03-11T22:50:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:21.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509470 2020-03-11T22:50:21.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:21.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:21.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509470 2020-03-11T22:50:21.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:21.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509470 2020-03-11T22:50:21.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2020-03-11T22:50:21.774 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509470 2020-03-11T22:50:21.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:22.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509470 -lt 64424509470 2020-03-11T22:50:22.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:22.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345948 2020-03-11T22:50:22.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:22.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:22.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345948 2020-03-11T22:50:22.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:22.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345948 2020-03-11T22:50:22.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345948' 2020-03-11T22:50:22.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:22.124 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345948 2020-03-11T22:50:22.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345949 -lt 85899345948 2020-03-11T22:50:22.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:22.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182427 2020-03-11T22:50:22.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:22.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:22.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182427 2020-03-11T22:50:22.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182427 2020-03-11T22:50:22.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2020-03-11T22:50:22.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:22.489 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182427 2020-03-11T22:50:22.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182428 -lt 107374182427 2020-03-11T22:50:22.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:22.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:22.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:22.835 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:22.836 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:22.836 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:22.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:22.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:22.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:22.922 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:22.923 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:22.923 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:22.923 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:22.923 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:22.923 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:22.923 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:22.923 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:22.924 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:22.925 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:22.925 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:22.925 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:22.925 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:22.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:50:23.413 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2020-03-11T22:50:23.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:50:25.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:25.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:25.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:25.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:25.779 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:50:25.779 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:50:25.779 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:50:25.780 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:50:25.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:25.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:25.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:25.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836515 2020-03-11T22:50:25.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515' 2020-03-11T22:50:25.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:25.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:26.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672995 2020-03-11T22:50:26.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995' 2020-03-11T22:50:26.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:26.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:26.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509473 2020-03-11T22:50:26.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995 2-64424509473' 2020-03-11T22:50:26.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:26.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:26.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345952 2020-03-11T22:50:26.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995 2-64424509473 3-85899345952' 2020-03-11T22:50:26.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:26.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:26.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182430 2020-03-11T22:50:26.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995 2-64424509473 3-85899345952 4-107374182430' 2020-03-11T22:50:26.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:26.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836515 2020-03-11T22:50:26.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:26.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:26.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836515 2020-03-11T22:50:26.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:26.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836515 2020-03-11T22:50:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2020-03-11T22:50:26.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:26.412 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836515 2020-03-11T22:50:26.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836515 -lt 21474836515 2020-03-11T22:50:26.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:26.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672995 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:26.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672995 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:26.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672995 2020-03-11T22:50:26.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2020-03-11T22:50:26.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:26.761 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672995 2020-03-11T22:50:27.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672995 -lt 42949672995 2020-03-11T22:50:27.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:27.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509473 2020-03-11T22:50:27.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:27.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:27.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509473 2020-03-11T22:50:27.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:27.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509473 2020-03-11T22:50:27.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2020-03-11T22:50:27.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:27.110 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509473 2020-03-11T22:50:27.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509471 -lt 64424509473 2020-03-11T22:50:27.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:28.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:28.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:28.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509473 -lt 64424509473 2020-03-11T22:50:28.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:28.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345952 2020-03-11T22:50:28.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:28.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:28.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345952 2020-03-11T22:50:28.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:28.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345952 2020-03-11T22:50:28.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2020-03-11T22:50:28.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:28.818 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345952 2020-03-11T22:50:29.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345952 -lt 85899345952 2020-03-11T22:50:29.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:29.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182430 2020-03-11T22:50:29.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:29.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:29.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182430 2020-03-11T22:50:29.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:29.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182430 2020-03-11T22:50:29.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182430' 2020-03-11T22:50:29.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:29.174 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182430 2020-03-11T22:50:29.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182431 -lt 107374182430 2020-03-11T22:50:29.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:29.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:29.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:29.522 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:29.522 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:29.522 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:29.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:29.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:29.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:29.609 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:29.609 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:29.609 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:29.609 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:29.610 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:50:29.610 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:29.610 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:50:29.610 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:29.610 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:29.610 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:50:29.610 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:29.611 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:29.612 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:29.613 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:29.613 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:29.613 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:29.613 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:29.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:29.952 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:29.953 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:50:18.613559+0000 94'150 104:214 [1,4] 1 [1] 1 0'0 2020-03-11T22:48:50.039875+0000 0'0 2020-03-11T22:48:50.039875+0000 0 2020-03-11T22:50:29.953 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:50:24.585840+0000 94'150 105:217 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:48:36.415824+0000 0'0 2020-03-11T22:48:36.415824+0000 0 2020-03-11T22:50:29.953 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:17.119773+0000 94'150 104:215 [1] 1 [1] 1 0'0 2020-03-11T22:48:41.034094+0000 0'0 2020-03-11T22:48:41.034094+0000 0 2020-03-11T22:50:29.953 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:50:29.953 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:29.954 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:50:29.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:50:30.587 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 2 2020-03-11T22:50:30.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:50:32.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:32.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:32.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:32.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:33.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836519 2020-03-11T22:50:33.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519' 2020-03-11T22:50:33.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:33.077 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:33.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672999 2020-03-11T22:50:33.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999' 2020-03-11T22:50:33.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:33.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:33.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509476 2020-03-11T22:50:33.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999 2-64424509476' 2020-03-11T22:50:33.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:33.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:33.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345955 2020-03-11T22:50:33.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999 2-64424509476 3-85899345955' 2020-03-11T22:50:33.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:33.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:33.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182434 2020-03-11T22:50:33.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999 2-64424509476 3-85899345955 4-107374182434' 2020-03-11T22:50:33.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:33.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836519 2020-03-11T22:50:33.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:33.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:33.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836519 2020-03-11T22:50:33.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:33.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836519 2020-03-11T22:50:33.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2020-03-11T22:50:33.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:33.583 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836519 2020-03-11T22:50:33.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836519 2020-03-11T22:50:33.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:34.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:34.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:35.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836519 -lt 21474836519 2020-03-11T22:50:35.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:35.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672999 2020-03-11T22:50:35.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:35.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:35.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672999 2020-03-11T22:50:35.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:35.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672999 2020-03-11T22:50:35.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2020-03-11T22:50:35.278 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:35.278 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672999 2020-03-11T22:50:35.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672999 -lt 42949672999 2020-03-11T22:50:35.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:35.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509476 2020-03-11T22:50:35.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:35.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:35.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509476 2020-03-11T22:50:35.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:35.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509476 2020-03-11T22:50:35.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2020-03-11T22:50:35.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:35.625 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509476 2020-03-11T22:50:35.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509477 -lt 64424509476 2020-03-11T22:50:35.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:35.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345955 2020-03-11T22:50:35.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:35.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:35.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345955 2020-03-11T22:50:35.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:35.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345955 2020-03-11T22:50:35.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345955' 2020-03-11T22:50:35.975 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345955 2020-03-11T22:50:35.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:36.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345956 -lt 85899345955 2020-03-11T22:50:36.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:36.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182434 2020-03-11T22:50:36.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:36.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:36.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182434 2020-03-11T22:50:36.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:36.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182434 2020-03-11T22:50:36.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182434' 2020-03-11T22:50:36.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:36.323 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182434 2020-03-11T22:50:36.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182434 -lt 107374182434 2020-03-11T22:50:36.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:36.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:36.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:36.693 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:36.693 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:36.693 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:36.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:36.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:36.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:36.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:36.787 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:36.787 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:36.788 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:50:36.789 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:50:36.789 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:36.789 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:36.789 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:36.789 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:36.789 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:36.789 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:50:36.790 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:36.790 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:36.790 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:50:36.790 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:50:36.790 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:36.790 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:36.790 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:36.791 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:36.791 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:50:36.791 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:36.791 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:36.791 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:36.791 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:50:36.791 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:36.792 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:36.793 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:36.793 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:36.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:37.108 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:37.109 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:50:18.613559+0000 94'150 108:218 [1,4] 1 [1] 1 0'0 2020-03-11T22:48:50.039875+0000 0'0 2020-03-11T22:48:50.039875+0000 0 2020-03-11T22:50:37.109 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:50:24.585840+0000 94'150 108:221 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:48:36.415824+0000 0'0 2020-03-11T22:48:36.415824+0000 0 2020-03-11T22:50:37.109 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-03-11T22:50:32.061962+0000 94'150 109:226 [1,2] 1 [1] 1 0'0 2020-03-11T22:48:41.034094+0000 0'0 2020-03-11T22:48:41.034094+0000 0 2020-03-11T22:50:37.109 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:50:37.109 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:37.110 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:50:37.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:37.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:50:37.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:50:37.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:50:37.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:37.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:37.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:50:37.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:50:37.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:50:37.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:37.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:37.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:37.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:37.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:37.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:50:37.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:50:37.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:50:37.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:50:37.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:50:39.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:39.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:39.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:39.465 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:39.465 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:39.466 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:39.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:39.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:39.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:39.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:39.572 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:39.572 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:39.572 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:39.572 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:39.572 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:50:39.572 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:39.572 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:50:39.573 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:39.573 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:39.573 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:50:39.573 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:50:39.573 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:39.573 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:39.573 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 254, 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:50:39.574 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:39.575 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:39.575 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:39.575 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:39.575 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:50:39.575 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:39.575 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:39.576 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:39.576 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:50:39.576 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:39.576 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:39.576 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:39.576 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:39.576 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:39.577 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:39.577 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:39.577 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:39.577 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:39.577 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:39.577 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:39.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:39.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:50:39.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:39.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:39.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:39.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:39.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:39.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:39.942 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:50:39.942 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:50:39.942 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:50:39.943 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:50:39.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:39.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:39.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:40.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836522 2020-03-11T22:50:40.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522' 2020-03-11T22:50:40.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:40.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:40.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673002 2020-03-11T22:50:40.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-42949673002' 2020-03-11T22:50:40.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:40.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:40.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509480 2020-03-11T22:50:40.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-42949673002 2-64424509480' 2020-03-11T22:50:40.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:40.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:40.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345959 2020-03-11T22:50:40.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-42949673002 2-64424509480 3-85899345959' 2020-03-11T22:50:40.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:40.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:40.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182437 2020-03-11T22:50:40.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-42949673002 2-64424509480 3-85899345959 4-107374182437' 2020-03-11T22:50:40.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:40.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836522 2020-03-11T22:50:40.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:40.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:40.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836522 2020-03-11T22:50:40.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:40.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836522 2020-03-11T22:50:40.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2020-03-11T22:50:40.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:40.577 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836522 2020-03-11T22:50:40.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836522 -lt 21474836522 2020-03-11T22:50:40.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:40.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673002 2020-03-11T22:50:40.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:40.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673002 2020-03-11T22:50:40.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673002 2020-03-11T22:50:40.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2020-03-11T22:50:40.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:40.922 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673002 2020-03-11T22:50:41.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673000 -lt 42949673002 2020-03-11T22:50:41.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:42.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:42.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:42.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673002 -lt 42949673002 2020-03-11T22:50:42.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:42.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509480 2020-03-11T22:50:42.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:42.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:42.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509480 2020-03-11T22:50:42.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:42.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509480 2020-03-11T22:50:42.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2020-03-11T22:50:42.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:42.626 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509480 2020-03-11T22:50:42.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509480 -lt 64424509480 2020-03-11T22:50:42.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:42.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345959 2020-03-11T22:50:42.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:42.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:42.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345959 2020-03-11T22:50:42.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:42.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345959 2020-03-11T22:50:42.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2020-03-11T22:50:42.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:42.974 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345959 2020-03-11T22:50:43.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345959 -lt 85899345959 2020-03-11T22:50:43.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:43.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182437 2020-03-11T22:50:43.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:43.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:43.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182437 2020-03-11T22:50:43.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:43.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182437 2020-03-11T22:50:43.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182437' 2020-03-11T22:50:43.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:43.326 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182437 2020-03-11T22:50:43.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182438 -lt 107374182437 2020-03-11T22:50:43.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:50:44.019 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:50:44.020 INFO:tasks.workunit.client.0.smithi130.stderr: 2020-03-11T22:50:44.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:50:46.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:46.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:46.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:46.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:46.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:46.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:46.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:46.136 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:46.136 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:46.136 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:46.137 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:46.137 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:50:46.137 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:46.137 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:50:46.137 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:46.137 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:46.138 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:50:46.139 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:46.139 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:46.139 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:46.139 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:50:46.139 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:46.139 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:46.139 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:46.140 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:46.140 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:50:46.140 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:46.141 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:46.141 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:50:46.141 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:50:46.141 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:46.141 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:46.141 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:46.141 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:46.142 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:46.142 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:46.142 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:46.142 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:46.142 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:46.142 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:46.142 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:46.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:46.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:50:46.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:50:46.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:50:46.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:46.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:46.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:50:46.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:50:46.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:50:46.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:46.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:46.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:46.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:46.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:50:46.478 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:50:46.479 INFO:tasks.workunit.client.0.smithi130.stderr: 2020-03-11T22:50:46.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:50:51.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:51.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:51.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:51.489 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:51.489 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:51.489 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:51.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:51.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:51.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:51.585 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:51.585 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:51.586 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:51.586 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:51.586 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:50:51.586 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:51.586 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:50:51.586 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:51.586 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:51.587 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:51.588 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:51.589 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:50:51.589 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:51.589 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:51.589 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:50:51.589 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:50:51.589 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:51.589 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:51.590 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:51.591 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:51.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:50:51.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:51.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:51.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:51.938 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:50:51.938 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:50:51.938 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:50:51.938 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:50:51.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:51.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:51.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:52.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836527 2020-03-11T22:50:52.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527' 2020-03-11T22:50:52.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:52.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:52.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673006 2020-03-11T22:50:52.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673006' 2020-03-11T22:50:52.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:52.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:52.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509484 2020-03-11T22:50:52.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673006 2-64424509484' 2020-03-11T22:50:52.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:52.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:52.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345963 2020-03-11T22:50:52.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673006 2-64424509484 3-85899345963' 2020-03-11T22:50:52.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:52.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:52.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182442 2020-03-11T22:50:52.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673006 2-64424509484 3-85899345963 4-107374182442' 2020-03-11T22:50:52.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:52.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836527 2020-03-11T22:50:52.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:52.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:52.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836527 2020-03-11T22:50:52.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:52.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836527 2020-03-11T22:50:52.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2020-03-11T22:50:52.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:52.570 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836527 2020-03-11T22:50:52.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836527 -lt 21474836527 2020-03-11T22:50:52.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:52.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673006 2020-03-11T22:50:52.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:52.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:52.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673006 2020-03-11T22:50:52.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:52.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673006 2020-03-11T22:50:52.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2020-03-11T22:50:52.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:52.917 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673006 2020-03-11T22:50:53.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673004 -lt 42949673006 2020-03-11T22:50:53.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:54.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:54.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:54.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673007 -lt 42949673006 2020-03-11T22:50:54.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:54.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509484 2020-03-11T22:50:54.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:54.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:54.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509484 2020-03-11T22:50:54.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:54.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509484 2020-03-11T22:50:54.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2020-03-11T22:50:54.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:54.625 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509484 2020-03-11T22:50:54.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509485 -lt 64424509484 2020-03-11T22:50:54.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:54.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345963 2020-03-11T22:50:54.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:54.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:54.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345963 2020-03-11T22:50:54.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:54.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345963 2020-03-11T22:50:54.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345963' 2020-03-11T22:50:54.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:54.984 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345963 2020-03-11T22:50:55.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345963 -lt 85899345963 2020-03-11T22:50:55.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:55.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182442 2020-03-11T22:50:55.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:55.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:55.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182442 2020-03-11T22:50:55.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:55.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182442 2020-03-11T22:50:55.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2020-03-11T22:50:55.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:55.356 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182442 2020-03-11T22:50:55.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182442 -lt 107374182442 2020-03-11T22:50:55.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:50:56.036 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:50:56.036 INFO:tasks.workunit.client.0.smithi130.stderr: 2020-03-11T22:50:56.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:50:58.046 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:58.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:58.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:58.047 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:58.047 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:58.047 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:58.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:58.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:50:58.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:58.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:58.152 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:50:58.153 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:50:58.153 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:58.153 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:58.153 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:50:58.153 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:58.153 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:50:58.154 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:58.154 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:58.154 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:50:58.154 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:50:58.154 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:58.154 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:58.154 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:50:58.155 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:58.156 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:58.156 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:58.156 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:58.156 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:50:58.156 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:50:58.156 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:50:58.156 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:50:58.157 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:50:58.157 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:50:58.157 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:58.157 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:50:58.157 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:50:58.157 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:50:58.157 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:50:58.158 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:50:58.158 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:50:58.158 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:50:58.158 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:50:58.158 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:50:58.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:58.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:50:58.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:50:58.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:50:58.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:58.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:58.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:50:58.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:50:58.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:50:58.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:50:58.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:58.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:50:58.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:50:58.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:50:58.867 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2020-03-11T22:50:58.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:50:59.627 INFO:tasks.workunit.client.0.smithi130.stderr:nobackfill is unset 2020-03-11T22:50:59.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:59.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:59.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:59.636 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:59.636 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:59.637 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:50:59.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:50:59.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:50:59.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:50:59.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:59.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:59.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:59.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:59.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:59.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:59.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:59.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:59.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:59.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:59.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:00.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:00.070 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:51:00.070 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:51:00.071 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:51:00.071 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2020-03-11T22:51:00.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:00.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:00.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:00.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836530 2020-03-11T22:51:00.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530' 2020-03-11T22:51:00.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:00.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:00.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673010 2020-03-11T22:51:00.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-42949673010' 2020-03-11T22:51:00.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:00.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:00.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509488 2020-03-11T22:51:00.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488' 2020-03-11T22:51:00.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:00.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:00.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345967 2020-03-11T22:51:00.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-85899345967' 2020-03-11T22:51:00.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:00.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:00.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182445 2020-03-11T22:51:00.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-42949673010 2-64424509488 3-85899345967 4-107374182445' 2020-03-11T22:51:00.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:00.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836530 2020-03-11T22:51:00.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:00.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:00.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836530 2020-03-11T22:51:00.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:00.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836530 2020-03-11T22:51:00.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2020-03-11T22:51:00.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:00.776 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836530 2020-03-11T22:51:01.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836530 2020-03-11T22:51:01.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:02.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:02.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:02.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836531 -lt 21474836530 2020-03-11T22:51:02.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:02.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673010 2020-03-11T22:51:02.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:02.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:02.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673010 2020-03-11T22:51:02.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:02.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673010 2020-03-11T22:51:02.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2020-03-11T22:51:02.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:02.542 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673010 2020-03-11T22:51:02.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673010 -lt 42949673010 2020-03-11T22:51:02.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:02.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509488 2020-03-11T22:51:02.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:02.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:02.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509488 2020-03-11T22:51:02.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:02.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509488 2020-03-11T22:51:02.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2020-03-11T22:51:02.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:02.977 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509488 2020-03-11T22:51:03.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509488 -lt 64424509488 2020-03-11T22:51:03.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:03.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345967 2020-03-11T22:51:03.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:03.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:03.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345967 2020-03-11T22:51:03.385 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:03.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345967 2020-03-11T22:51:03.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2020-03-11T22:51:03.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:03.386 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345967 2020-03-11T22:51:03.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345967 -lt 85899345967 2020-03-11T22:51:03.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:03.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182445 2020-03-11T22:51:03.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:03.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:03.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182445 2020-03-11T22:51:03.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:03.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182445 2020-03-11T22:51:03.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2020-03-11T22:51:03.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:03.737 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182445 2020-03-11T22:51:04.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182446 -lt 107374182445 2020-03-11T22:51:04.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:51:04.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:04.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:04.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:51:04.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:04.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:04.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:04.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:04.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:04.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:04.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:04.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:04.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:04.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:04.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:05.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:05.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:51:05.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:05.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:51:05.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:05.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:05.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:05.534 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:05.534 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:05.535 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:05.535 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:05.535 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:05.535 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:05.535 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:05.535 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:05.535 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:05.536 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:05.537 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:05.537 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:05.537 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:05.537 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:05.537 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:05.537 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:05.537 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:05.538 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:05.539 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:05.539 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:05.539 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:05.539 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:05.539 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:05.539 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:05.539 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:05.540 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:05.540 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:05.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:05.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:05.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:05.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:05.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:05.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:05.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:05.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:05.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:05.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:05.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:05.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:05.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:06.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:06.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:06.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:06.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:06.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:06.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:06.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:06.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:06.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=7211871 2020-03-11T22:51:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 7211871 '!=' null 2020-03-11T22:51:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:06.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:06.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:07.043 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:07.043 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:07.043 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:07.043 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:07.043 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:07.043 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:07.044 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:07.044 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:07.044 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:07.044 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:07.044 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:07.044 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:07.045 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:07.045 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:07.045 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:07.045 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:07.045 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:07.045 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:07.045 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:07.046 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:07.048 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:07.048 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:07.048 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:07.048 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:07.048 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:07.048 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:07.048 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:07.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:07.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:07.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:07.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:07.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:07.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:07.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:07.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:07.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:07.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:07.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:07.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:07.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:07.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:07.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:07.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:07.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:07.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:07.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:07.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:07.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:07.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:08.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20231157 2020-03-11T22:51:08.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20231157 '!=' null 2020-03-11T22:51:08.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:08.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:08.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:08.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:08.630 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:08.631 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:08.631 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:08.631 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:08.631 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:08.631 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:08.631 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:08.632 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:08.632 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:08.632 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:08.632 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:08.632 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:08.632 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:08.632 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:08.633 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:08.633 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:08.633 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:08.633 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:08.633 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:08.633 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:08.633 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:08.634 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:08.634 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:08.634 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:08.634 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:08.634 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:08.634 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:08.634 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:08.635 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:08.635 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:08.635 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:08.635 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:08.635 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:08.635 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:08.635 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:08.636 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:08.636 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:08.636 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:08.636 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:08.636 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:08.636 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:08.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:08.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:08.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:08.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:08.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:08.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:08.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:08.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:08.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:09.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:09.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:09.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:09.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:09.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:09.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:09.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:09.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:09.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:09.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:09.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:09.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:09.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:10.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20231157 2020-03-11T22:51:10.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20231157 '!=' null 2020-03-11T22:51:10.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:10.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:10.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:10.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:10.109 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:10.109 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:10.109 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:10.109 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:10.109 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:10.110 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:10.110 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:10.110 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:10.110 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:10.110 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:10.110 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:10.110 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:10.111 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:10.112 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:10.112 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:10.112 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:10.112 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:10.112 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:10.112 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:10.112 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:10.113 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:10.114 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:10.114 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:10.114 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:10.114 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:10.114 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:10.114 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:10.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:10.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:10.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:10.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:10.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:10.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:10.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:10.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:10.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:10.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:10.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:10.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:10.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:11.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:11.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:11.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:11.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:11.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:11.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:11.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:11.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:11.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:11.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=16959199 2020-03-11T22:51:11.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 16959199 '!=' null 2020-03-11T22:51:11.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:11.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:11.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:11.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:11.540 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:11.541 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:11.541 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:11.541 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:11.541 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:11.541 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:11.541 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:11.542 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:11.542 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:11.542 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:11.542 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:11.542 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:11.542 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:11.542 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:11.543 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:11.544 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:11.544 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:11.544 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:11.544 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:11.544 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:11.544 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:11.544 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:11.545 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:11.545 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:11.545 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:11.545 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:11.545 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:11.545 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:11.545 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:11.546 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:11.546 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:11.553 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:11.554 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:11.554 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:11.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:11.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:11.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:11.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:11.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:11.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:11.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:11.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:11.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:11.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:11.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:11.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:11.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:12.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:12.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:12.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:12.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:12.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:12.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14851862 2020-03-11T22:51:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14851862 '!=' null 2020-03-11T22:51:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:12.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:12.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:12.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:13.033 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:13.033 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:13.034 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:13.035 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:13.035 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:13.035 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:13.035 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:13.035 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:13.035 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:13.035 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:13.036 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:13.037 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:13.037 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:13.037 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:13.037 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:13.037 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:13.037 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:13.037 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:13.038 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:13.038 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:13.038 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:13.038 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:13.038 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:13.038 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:13.039 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:13.039 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:13.039 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:13.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:13.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:13.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:13.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:13.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:13.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:13.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:13.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:13.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:13.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:13.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:13.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:13.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:13.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:13.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:13.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:14.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26209168 2020-03-11T22:51:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26209168 '!=' null 2020-03-11T22:51:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:14.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:14.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:14.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:14.483 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:14.484 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:14.484 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:14.484 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:14.484 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:14.484 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:14.484 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:14.484 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:14.485 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:14.485 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:14.485 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:14.485 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:14.485 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:14.485 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:14.485 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:14.486 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:14.486 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:14.486 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:14.486 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:14.486 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:14.486 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:14.486 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:14.487 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:14.488 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:14.488 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:14.488 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:14.488 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:14.488 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:14.488 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:14.488 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:14.489 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:14.489 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:14.489 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:14.489 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:14.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:14.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:14.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:14.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:14.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:14.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:14.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:14.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:14.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:14.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:14.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:14.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:14.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:15.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:15.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:15.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:15.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:15.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:15.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:15.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:15.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:15.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:15.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26209168 2020-03-11T22:51:15.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26209168 '!=' null 2020-03-11T22:51:15.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:15.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:15.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:15.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:15.978 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:15.978 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:15.978 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:15.978 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:15.979 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:15.980 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:15.981 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:15.981 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:15.981 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:15.981 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:15.981 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:15.981 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:15.981 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:15.982 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:15.982 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:15.982 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:15.982 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:15.982 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:15.982 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:15.982 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:15.983 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:15.983 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:15.983 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:15.983 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:15.983 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:15.983 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:15.983 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:15.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:16.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:16.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:16.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:16.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:16.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:16.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:16.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:16.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:16.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:16.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:16.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:16.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:16.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:16.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:16.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:16.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:16.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:16.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:16.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:16.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:17.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841218 2020-03-11T22:51:17.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841218 '!=' null 2020-03-11T22:51:17.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:17.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:17.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:17.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:17.423 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:17.423 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:17.423 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:17.423 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:17.424 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:17.424 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:17.424 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:17.424 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:17.424 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:17.424 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:17.424 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:17.425 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:17.426 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:17.426 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:17.426 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:17.426 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:17.426 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:17.426 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:17.426 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:17.427 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:17.428 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:17.428 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:17.428 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:17.428 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:17.428 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:17.428 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:17.428 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:17.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:17.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:17.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:17.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:17.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:17.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:17.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:17.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:17.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:17.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:17.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:17.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:17.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:18.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:18.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:18.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:18.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:18.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:18.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:18.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:18.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:18.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198673 2020-03-11T22:51:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198673 '!=' null 2020-03-11T22:51:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:18.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:18.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:18.932 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:18.932 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:18.932 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:18.932 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:18.932 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:18.933 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:18.934 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:18.934 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:18.934 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:18.934 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:18.934 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:18.934 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:18.934 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:18.935 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:18.936 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:18.937 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:18.937 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:18.937 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:18.937 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:18.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:19.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:19.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:19.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:19.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:19.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:19.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:19.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:19.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:19.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:19.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:19.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:19.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:19.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:19.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:19.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:19.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:19.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:19.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:19.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:19.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:19.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:20.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198673 2020-03-11T22:51:20.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198673 '!=' null 2020-03-11T22:51:20.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:20.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:20.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:20.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:20.418 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:20.418 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:20.419 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:20.419 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:20.419 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:20.419 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:20.419 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:20.419 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:20.419 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:20.420 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2020-03-11T22:51:20.421 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:20.421 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:20.421 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:20.421 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:20.421 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:20.421 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:20.421 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:20.422 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:20.423 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:20.423 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:20.423 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:20.423 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:20.423 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:20.423 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:20.423 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:20.424 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:20.424 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:20.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:20.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:20.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:20.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:20.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:20.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:20.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:20.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:20.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:20.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:20.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:20.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:20.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:21.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:21.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:21.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:21.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:21.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:21.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:21.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:21.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:21.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22714765 2020-03-11T22:51:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22714765 '!=' null 2020-03-11T22:51:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:21.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:21.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:21.922 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:21.922 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:21.922 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:21.922 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:21.922 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:21.922 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:21.923 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:21.923 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:21.923 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:21.923 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:21.923 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:21.923 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:21.923 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:21.924 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:21.924 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:21.924 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:21.924 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:21.924 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:21.924 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:21.924 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:21.925 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:21.925 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:21.925 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:21.926 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:21.926 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:21.926 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:21.926 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:21.926 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:21.926 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:21.926 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:21.927 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:21.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:22.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:22.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:22.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:22.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:22.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:22.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:22.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:22.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:22.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:22.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:22.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:22.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:22.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:22.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:22.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:22.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:22.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:22.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:22.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:22.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:22.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:23.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036301 2020-03-11T22:51:23.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036301 '!=' null 2020-03-11T22:51:23.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:23.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:23.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:23.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:23.383 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:23.383 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:23.383 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:23.383 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:23.383 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:23.384 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:23.385 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:23.385 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:23.385 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:23.385 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:23.385 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:23.385 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:23.385 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:23.386 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:23.386 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:23.386 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:23.386 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:23.386 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:23.386 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:23.386 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:23.387 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:23.387 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:23.388 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:23.388 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:23.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:23.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:23.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:23.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:23.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:23.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:23.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:23.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:23.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:23.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:23.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:23.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:23.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:24.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:51:24.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:24.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:24.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:24.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:24.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:24.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:24.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:24.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:24.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451852 2020-03-11T22:51:24.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451852 '!=' null 2020-03-11T22:51:24.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:24.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:24.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:24.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:24.943 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:24.943 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:24.943 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:24.943 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:24.943 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:24.943 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:24.944 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:24.944 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:24.944 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:24.944 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:24.944 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:24.944 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:24.944 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:24.945 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:24.945 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:24.945 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:24.945 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:24.945 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:24.945 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:24.945 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:24.946 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:24.946 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:24.946 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:24.946 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:24.946 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:24.946 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:24.946 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:24.947 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:24.947 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:24.947 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:24.947 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:24.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:25.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:25.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:25.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:25.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:25.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:25.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:25.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:25.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:25.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:25.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:25.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:25.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:25.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:25.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:51:25.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:25.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:51:25.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:25.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:25.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:26.003 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:26.003 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:26.003 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:26.003 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:26.003 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:26.003 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:26.004 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:26.004 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:26.004 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:26.004 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:26.004 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:26.004 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:26.004 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:26.005 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:26.006 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:26.006 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:26.006 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:26.006 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:26.006 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:26.006 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:26.007 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:26.007 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:26.007 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:26.007 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:26.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:26.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:26.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:26.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:26.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:26.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:26.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:26.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:26.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:26.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:26.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:26.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:26.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:26.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:26.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:26.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:26.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:26.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:26.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:26.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:26.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:26.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:27.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415237 2020-03-11T22:51:27.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415237 '!=' null 2020-03-11T22:51:27.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:27.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:27.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:27.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:27.452 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:27.452 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:27.452 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:27.452 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:27.452 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:27.452 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:27.453 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:27.453 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:27.453 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:27.453 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:27.453 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:27.453 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:27.453 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:27.454 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:27.455 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:27.455 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:27.455 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:27.455 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:27.455 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:27.455 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:27.455 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:27.456 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:27.456 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:27.456 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:27.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:27.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:27.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:27.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:27.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:27.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:27.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:27.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:27.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:28.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:28.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:28.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:28.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:28.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:28.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:28.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:28.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:28.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:28.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451516 2020-03-11T22:51:28.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451516 '!=' null 2020-03-11T22:51:28.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:28.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:28.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:28.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:29.040 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:29.040 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:29.040 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:29.040 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:29.040 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:29.040 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:29.041 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:29.041 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:29.041 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:29.041 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:29.041 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:29.041 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:29.041 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:29.042 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:29.042 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:29.042 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:29.042 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:29.042 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:29.042 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:29.042 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:29.043 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:29.044 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:29.044 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:29.044 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:29.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:29.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:29.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:29.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:29.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:29.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:29.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:29.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:29.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:29.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:29.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:29.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:29.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:29.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:29.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:29.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:29.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:29.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:29.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:29.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:29.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:29.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:30.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27026300 2020-03-11T22:51:30.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27026300 '!=' null 2020-03-11T22:51:30.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:30.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:30.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:30.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:30.559 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:30.560 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:30.560 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:30.560 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:30.560 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:30.560 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:30.560 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:30.560 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:30.561 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:30.561 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:30.561 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:30.561 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:30.561 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:30.561 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:30.562 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:30.562 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:30.562 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:30.562 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:30.562 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:30.562 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:30.562 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:30.563 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:30.564 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:30.564 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:30.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:30.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:30.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:30.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:30.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:30.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:30.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:31.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:31.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:31.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:31.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:31.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:31.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:31.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27026300 2020-03-11T22:51:31.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27026300 '!=' null 2020-03-11T22:51:31.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:31.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:31.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:31.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:32.024 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:32.025 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:32.025 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:32.025 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:32.025 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:32.025 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:32.025 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:32.025 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:32.026 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:32.027 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:32.028 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:32.028 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:32.028 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:32.028 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:32.028 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:32.028 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:32.028 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:32.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:32.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:32.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:32.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:32.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:32.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:32.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:32.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:32.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:32.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:32.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:32.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:32.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:32.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:32.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:33.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22871643 2020-03-11T22:51:33.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22871643 '!=' null 2020-03-11T22:51:33.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:33.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:33.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:33.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:33.494 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:33.494 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:33.494 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:33.494 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:33.494 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:33.495 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:33.495 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:33.495 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:33.495 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:33.495 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:33.495 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:33.495 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:33.496 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:33.496 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:33.496 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:33.496 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:33.496 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:33.496 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:33.496 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:33.497 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:33.498 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:33.498 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:33.498 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:33.498 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:33.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:33.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:33.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:33.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:33.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:33.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:33.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:33.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:33.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:33.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:33.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:33.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:33.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:34.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:34.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:34.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:34.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:34.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:34.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:34.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32325327 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32325327 '!=' null 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:34.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:34.962 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:34.962 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:34.962 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:34.962 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:34.962 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:34.962 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:34.963 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:34.963 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:34.963 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:34.963 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:34.963 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:34.963 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:34.963 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:34.964 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:34.965 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:34.966 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:34.966 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:34.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:35.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:35.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:35.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:35.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:35.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:35.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:35.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:35.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:35.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:35.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:35.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:35.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:35.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:35.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:35.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:35.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:35.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:35.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:36.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32325327 2020-03-11T22:51:36.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32325327 '!=' null 2020-03-11T22:51:36.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:36.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:36.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:36.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:36.443 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:36.443 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:36.443 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:36.443 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:36.443 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:36.443 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:36.444 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:36.444 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:36.444 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:36.444 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:36.444 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:36.444 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:36.444 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:36.445 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:36.445 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:36.445 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:36.445 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:36.445 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:36.445 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:36.445 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:36.446 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:36.446 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:36.446 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:36.446 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:36.446 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:36.446 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:36.446 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:36.447 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:36.447 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:36.447 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:36.447 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:36.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:36.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:36.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:36.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:36.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:36.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:36.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:36.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:36.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:36.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:36.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:36.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:36.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:37.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:37.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:37.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:37.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:37.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:37.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:37.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:37.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:37.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:37.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22715063 2020-03-11T22:51:37.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22715063 '!=' null 2020-03-11T22:51:37.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:37.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:37.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:37.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:37.998 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:37.999 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:38.000 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:38.001 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:38.001 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:38.001 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:38.001 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:38.001 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:38.001 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:38.001 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:38.002 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:38.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:38.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:38.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:38.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:38.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:38.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:38.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:38.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:38.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:38.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:38.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:38.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:38.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:38.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:38.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:38.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:38.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:38.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:38.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:38.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:38.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:38.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:39.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34073017 2020-03-11T22:51:39.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34073017 '!=' null 2020-03-11T22:51:39.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:39.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:39.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:39.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:39.499 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:39.499 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:39.499 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:39.500 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:39.500 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2020-03-11T22:51:39.500 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:39.500 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2020-03-11T22:51:39.500 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2020-03-11T22:51:39.500 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:39.500 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:39.501 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:39.502 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2020-03-11T22:51:39.502 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2020-03-11T22:51:39.502 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:39.502 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:39.502 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:39.502 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:39.502 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:39.503 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:39.503 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:39.503 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:39.503 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:39.503 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:39.503 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:39.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:39.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:39.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:39.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:39.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:39.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:39.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:39.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:39.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:39.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:39.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:39.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:39.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:40.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:40.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:40.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:40.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:40.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:40.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:40.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:40.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:40.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22715311 2020-03-11T22:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22715311 '!=' null 2020-03-11T22:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:41.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:41.348 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:41.351 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:41.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:41.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:41.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:41.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:41.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:41.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:41.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:41.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:41.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:41.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:41.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:41.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:41.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:42.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:42.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:42.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:42.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:42.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:42.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:42.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:42.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:42.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:42.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036508 2020-03-11T22:51:42.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036508 '!=' null 2020-03-11T22:51:42.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:42.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:42.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:42.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:42.900 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:42.900 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:42.900 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:42.900 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:42.900 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:42.901 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:42.901 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:42.901 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:42.901 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:42.901 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:42.901 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:42.901 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:42.903 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:42.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:43.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:43.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:43.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:43.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:43.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:43.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:43.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:43.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:43.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:43.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:43.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:43.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:43.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:51:43.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:43.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:43.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:43.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:43.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:43.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:43.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:43.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:44.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451954 2020-03-11T22:51:44.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451954 '!=' null 2020-03-11T22:51:44.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:44.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:44.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:44.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:44.467 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:44.467 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:44.467 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:44.467 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:44.467 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:44.467 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:44.468 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:44.469 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:44.469 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:44.469 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:44.469 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:44.469 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:44.469 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:44.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:44.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:44.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:44.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:44.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:44.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:44.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:44.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:44.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:44.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:44.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:44.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:45.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:45.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-03-11T22:51:45.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:45.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-03-11T22:51:45.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:45.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:45.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:45.478 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:45.478 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:45.478 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:45.479 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:45.479 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:45.479 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:45.479 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:45.479 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:45.479 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:45.479 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:45.480 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:45.481 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:45.481 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:45.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:45.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:45.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:45.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:45.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:45.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:45.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:45.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:45.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:45.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:45.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:45.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:45.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:46.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:46.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:46.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:46.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:46.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:46.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:46.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:46.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:46.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:46.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415420 2020-03-11T22:51:46.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415420 '!=' null 2020-03-11T22:51:46.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:46.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:46.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:46.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:46.937 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:46.937 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:46.937 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:46.938 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:46.938 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:46.938 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:46.938 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:46.938 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:46.938 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:46.938 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:46.939 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:46.939 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:46.939 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:46.939 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:46.939 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:46.939 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:46.939 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:46.940 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:46.940 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:46.940 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:46.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:47.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:47.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:47.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:47.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:47.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:47.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:47.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:47.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:47.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:47.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:47.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:47.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:47.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:47.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:47.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:47.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:47.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:47.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:47.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:47.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:47.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:48.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415420 2020-03-11T22:51:48.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415420 '!=' null 2020-03-11T22:51:48.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:48.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:48.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:48.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:48.373 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:48.373 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:48.373 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:48.373 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:48.374 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:48.375 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:48.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:48.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:48.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:48.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:48.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:48.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:48.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:48.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:48.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:48.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:48.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:48.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:48.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:49.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:49.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:49.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31452061 2020-03-11T22:51:49.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31452061 '!=' null 2020-03-11T22:51:49.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:49.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:49.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:49.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:49.892 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:49.892 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:49.892 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:49.893 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:49.894 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:49.894 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:49.894 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:49.894 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:49.894 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:49.894 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:49.894 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:49.895 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:49.895 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:49.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:50.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:50.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:50.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:50.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:50.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:50.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:50.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:50.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:50.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:50.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:50.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:50.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:50.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:50.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:50.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:50.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:50.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:50.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:51.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26428491 2020-03-11T22:51:51.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26428491 '!=' null 2020-03-11T22:51:51.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:51.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:51.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:51.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:51.340 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:51.340 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:51.341 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:51.341 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:51.341 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:51.341 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:51.341 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:51.341 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:51.341 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:51.342 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:51.342 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:51.342 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:51.342 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:51.342 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:51.342 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:51.342 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:51.343 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:51.343 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:51.343 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:51.343 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:51.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:51.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:51.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:51.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:51.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:51.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:51.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:51.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:51.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:51.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:51.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:51.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:51.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:52.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:52.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:52.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:52.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:52.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:52.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:52.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:52.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:52.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:53.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22827561 2020-03-11T22:51:53.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22827561 '!=' null 2020-03-11T22:51:53.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:53.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:53.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:53.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:53.199 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:53.199 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:53.199 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:53.199 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:53.200 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:53.200 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:53.200 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:53.200 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:53.200 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:53.200 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:53.200 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:53.201 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:53.201 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:53.201 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:53.201 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:53.201 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:53.201 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:53.201 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:53.202 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:53.202 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:53.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:53.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:53.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:53.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:53.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:53.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:53.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:53.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:53.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:53.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:53.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:53.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:53.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:54.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:54.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:54.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:54.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:54.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:54.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:54.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:54.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:54.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:54.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578472 2020-03-11T22:51:54.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578472 '!=' null 2020-03-11T22:51:54.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:54.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:54.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:54.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:54.670 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:54.670 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:54.670 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:54.671 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:54.671 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:54.671 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:54.671 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:54.671 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:54.671 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:54.671 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:54.672 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:54.673 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:54.673 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:54.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:54.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:54.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:54.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:54.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:54.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:54.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:54.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:55.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:55.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:55.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:55.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:55.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:55.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:56.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578472 2020-03-11T22:51:56.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578472 '!=' null 2020-03-11T22:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:56.119 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:56.119 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:56.119 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:56.119 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:56.119 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:56.119 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:56.121 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:56.121 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:56.121 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:56.121 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:56.121 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:56.121 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:56.121 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:56.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:56.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:56.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:56.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:56.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:56.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:56.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:56.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:56.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:56.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:56.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:56.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:56.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:57.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:57.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:57.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967962 2020-03-11T22:51:57.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967962 '!=' null 2020-03-11T22:51:57.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:57.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:57.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.619 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:57.620 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:57.620 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:57.620 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:57.620 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:57.620 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:57.620 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:57.620 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:57.621 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:57.621 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:57.621 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:57.621 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:57.621 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:57.621 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:57.621 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:57.622 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:57.622 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:57.622 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:57.622 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:57.622 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:57.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:57.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:57.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:57.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:57.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:57.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:57.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:57.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:57.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:58.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:58.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:58.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:58.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:58.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:58.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:58.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:58.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:58.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:58.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:58.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:58.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:58.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:58.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32325558 2020-03-11T22:51:58.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32325558 '!=' null 2020-03-11T22:51:58.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:58.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:58.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:58.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:59.060 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:51:59.060 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:51:59.061 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:59.061 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:59.061 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:59.061 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:51:59.061 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:51:59.061 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:51:59.062 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:51:59.062 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:51:59.062 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:59.062 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:51:59.062 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:51:59.063 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:51:59.063 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:51:59.063 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:51:59.063 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:51:59.063 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:51:59.063 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:51:59.063 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:51:59.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:59.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:59.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:59.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:59.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:59.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:59.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:59.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:59.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:59.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:59.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:59.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:59.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:59.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:59.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:59.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:59.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:59.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:59.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:59.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:00.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967860 2020-03-11T22:52:00.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967860 '!=' null 2020-03-11T22:52:00.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:00.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:00.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:00.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:00.564 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:52:00.564 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:52:00.564 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:00.564 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:00.564 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:00.564 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:00.565 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:52:00.566 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:00.566 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:00.566 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:00.566 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:52:00.566 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:00.566 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:52:00.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:00.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:00.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:00.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:00.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:00.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:00.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:00.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:00.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:01.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:01.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:01.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:01.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:01.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:01.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:01.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:01.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:01.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:01.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:01.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:01.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:01.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:01.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967860 2020-03-11T22:52:01.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967860 '!=' null 2020-03-11T22:52:01.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:01.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:01.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:01.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:02.041 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:52:02.041 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:52:02.041 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:02.042 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:02.042 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:02.042 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:52:02.042 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:02.042 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:52:02.042 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:02.042 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:02.043 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:02.043 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:52:02.043 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:02.043 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:52:02.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:02.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:02.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:02.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:02.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:02.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:02.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:02.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:02.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:02.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:02.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:02.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:02.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:02.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:02.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:02.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:02.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:02.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:02.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:02.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:02.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:02.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:03.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036287 2020-03-11T22:52:03.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036287 '!=' null 2020-03-11T22:52:03.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:03.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:03.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:03.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:03.471 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:52:03.471 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:52:03.472 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:03.472 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:03.472 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:03.472 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:52:03.472 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:03.472 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:52:03.473 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:03.473 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:03.473 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:03.473 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:52:03.473 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:03.473 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:52:03.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:03.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:03.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:03.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:03.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:03.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:03.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:03.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:03.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:03.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:03.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:03.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:03.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:04.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831070 2020-03-11T22:52:04.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831070 '!=' null 2020-03-11T22:52:04.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:04.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:04.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:04.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:04.930 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:52:04.930 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2020-03-11T22:52:04.930 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:04.930 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:04.930 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2020-03-11T22:52:04.931 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:04.932 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:52:04.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:05.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:05.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:05.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:05.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:05.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:05.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:05.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:05.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:05.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:52:05.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:05.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:05.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:52:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:52:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:52:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:52:06.179 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:52:06.180 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:21.283014+0000 94'150 118:342 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:48:50.039875+0000 0'0 2020-03-11T22:48:50.039875+0000 0 2020-03-11T22:52:06.180 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:01.833053+0000 94'150 119:342 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:36.415824+0000 0'0 2020-03-11T22:48:36.415824+0000 0 2020-03-11T22:52:06.180 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:41.335540+0000 94'150 118:352 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:48:41.034094+0000 0'0 2020-03-11T22:48:41.034094+0000 0 2020-03-11T22:52:06.180 INFO:tasks.workunit.client.0.smithi130.stdout: 2020-03-11T22:52:06.180 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:52:06.181 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:52:06.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:06.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:06.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:06.191 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:06.191 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:06.191 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:52:06.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56067/ceph-osd.1.asok 2020-03-11T22:52:06.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:52:06.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56067/ceph-osd.1.asok dump_pgstate_history 2020-03-11T22:52:06.280 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:52:06.280 INFO:tasks.workunit.client.0.smithi130.stdout: "pgs": [ 2020-03-11T22:52:06.280 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.281 INFO:tasks.workunit.client.0.smithi130.stdout: "pg": "7.0", 2020-03-11T22:52:06.281 INFO:tasks.workunit.client.0.smithi130.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.281 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [ 2020-03-11T22:52:06.281 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.281 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "62", 2020-03-11T22:52:06.282 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.282 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.282 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.282 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.417324+0000", 2020-03-11T22:52:06.282 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:50.417335+0000" 2020-03-11T22:52:06.282 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.283 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.283 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.283 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.417342+0000", 2020-03-11T22:52:06.283 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:50.439405+0000" 2020-03-11T22:52:06.283 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.283 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.283 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.439407+0000", 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:50.439473+0000" 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.439473+0000", 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:50.439487+0000" 2020-03-11T22:52:06.284 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.285 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.285 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.285 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.439488+0000", 2020-03-11T22:52:06.285 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:50.679492+0000" 2020-03-11T22:52:06.285 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.285 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.285 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.286 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.417339+0000", 2020-03-11T22:52:06.286 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:50.679500+0000" 2020-03-11T22:52:06.286 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.286 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.286 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.286 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.679680+0000", 2020-03-11T22:52:06.286 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:51.278012+0000" 2020-03-11T22:52:06.287 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.287 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.287 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.287 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:51.278013+0000", 2020-03-11T22:52:06.287 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:51.278080+0000" 2020-03-11T22:52:06.287 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.287 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.288 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.288 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:51.278080+0000", 2020-03-11T22:52:06.288 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.054970+0000" 2020-03-11T22:52:06.288 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.288 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.288 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.289 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.679500+0000", 2020-03-11T22:52:06.289 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.054988+0000" 2020-03-11T22:52:06.289 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.289 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.289 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.289 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.417336+0000", 2020-03-11T22:52:06.290 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.055005+0000" 2020-03-11T22:52:06.290 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.290 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.290 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.290 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:50.417321+0000", 2020-03-11T22:52:06.290 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.055008+0000" 2020-03-11T22:52:06.290 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.291 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.291 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.291 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.291 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "63", 2020-03-11T22:52:06.291 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.291 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.291 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.292 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.055009+0000", 2020-03-11T22:52:06.292 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.055237+0000" 2020-03-11T22:52:06.292 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.292 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.292 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.292 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.292 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "92", 2020-03-11T22:52:06.293 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.293 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.293 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.293 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.055241+0000", 2020-03-11T22:52:06.293 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.055251+0000" 2020-03-11T22:52:06.293 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.055258+0000", 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.090281+0000" 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.294 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.090282+0000", 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.090394+0000" 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.090395+0000", 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.090420+0000" 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.295 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.090420+0000", 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.221154+0000" 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.055255+0000", 2020-03-11T22:52:06.296 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.221162+0000" 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.221297+0000", 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.360486+0000" 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.297 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.360487+0000", 2020-03-11T22:52:06.298 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:52.360537+0000" 2020-03-11T22:52:06.298 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.298 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.298 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.298 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.360537+0000", 2020-03-11T22:52:06.298 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.778445+0000" 2020-03-11T22:52:06.298 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.299 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.299 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.299 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.221163+0000", 2020-03-11T22:52:06.299 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.778463+0000" 2020-03-11T22:52:06.299 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.055252+0000", 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.778477+0000" 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.300 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:52.055237+0000", 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.778481+0000" 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "92", 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.301 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.302 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.302 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.778481+0000", 2020-03-11T22:52:06.302 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.778656+0000" 2020-03-11T22:52:06.302 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.302 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.302 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.302 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "99", 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.778660+0000", 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.778672+0000" 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.303 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.304 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.304 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.778679+0000", 2020-03-11T22:52:06.304 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.928346+0000" 2020-03-11T22:52:06.304 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.304 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.304 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.304 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.928347+0000", 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.928404+0000" 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.928405+0000", 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.928416+0000" 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.305 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.306 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.306 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.928417+0000", 2020-03-11T22:52:06.306 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.045330+0000" 2020-03-11T22:52:06.306 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.778676+0000", 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.045338+0000" 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.307 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.045398+0000", 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.070011+0000" 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.070011+0000", 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.070058+0000" 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.308 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.309 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.309 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.070058+0000", 2020-03-11T22:52:06.309 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.792470+0000" 2020-03-11T22:52:06.309 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.309 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.309 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.309 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.045338+0000", 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.792489+0000" 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.778673+0000", 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.792527+0000" 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.310 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.778656+0000", 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.792532+0000" 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.311 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "99", 2020-03-11T22:52:06.312 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.312 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.312 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.312 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.792532+0000", 2020-03-11T22:52:06.312 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.792804+0000" 2020-03-11T22:52:06.312 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.312 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.313 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.313 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.313 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "101", 2020-03-11T22:52:06.313 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.313 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.792809+0000", 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.792824+0000" 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.792832+0000", 2020-03-11T22:52:06.314 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.793363+0000" 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.793363+0000", 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.793419+0000" 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.315 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.792828+0000", 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:13.793429+0000" 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.793430+0000", 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.101783+0000" 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.316 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.792824+0000", 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.101804+0000" 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:13.792804+0000", 2020-03-11T22:52:06.317 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.101810+0000" 2020-03-11T22:52:06.318 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.318 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.318 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.318 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.318 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "101", 2020-03-11T22:52:06.318 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.318 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.101810+0000", 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.102045+0000" 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.319 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "114", 2020-03-11T22:52:06.320 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.320 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.320 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.320 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.102051+0000", 2020-03-11T22:52:06.320 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.102083+0000" 2020-03-11T22:52:06.320 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.102093+0000", 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.235575+0000" 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.321 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.235576+0000", 2020-03-11T22:52:06.322 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.235655+0000" 2020-03-11T22:52:06.322 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.322 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.322 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.322 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.235655+0000", 2020-03-11T22:52:06.322 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.235676+0000" 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.235677+0000", 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.291857+0000" 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.323 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.102088+0000", 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.291871+0000" 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.292140+0000", 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.818547+0000" 2020-03-11T22:52:06.324 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.818548+0000", 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.818696+0000" 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:06.325 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.818696+0000", 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:14.819083+0000" 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.819084+0000", 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:43.923525+0000" 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.326 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.327 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:52:06.327 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:43.923525+0000", 2020-03-11T22:52:06.327 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:43.923864+0000" 2020-03-11T22:52:06.327 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.327 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.327 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:06.327 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:43.923864+0000", 2020-03-11T22:52:06.328 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:55.938609+0000" 2020-03-11T22:52:06.328 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.328 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.328 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:06.328 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:55.938610+0000", 2020-03-11T22:52:06.328 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:55.938986+0000" 2020-03-11T22:52:06.328 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.329 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.329 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:06.329 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:55.938986+0000", 2020-03-11T22:52:06.329 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.700964+0000" 2020-03-11T22:52:06.329 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.329 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.330 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.330 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:20.700965+0000", 2020-03-11T22:52:06.330 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.701137+0000" 2020-03-11T22:52:06.330 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.330 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.331 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.331 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:20.701137+0000", 2020-03-11T22:52:06.331 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.849169+0000" 2020-03-11T22:52:06.331 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.331 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.331 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.331 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.291872+0000", 2020-03-11T22:52:06.332 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.849200+0000" 2020-03-11T22:52:06.332 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.332 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.332 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.332 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.102084+0000", 2020-03-11T22:52:06.332 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.849221+0000" 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:14.102045+0000", 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.849227+0000" 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.333 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.334 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.334 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "114", 2020-03-11T22:52:06.334 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.334 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.334 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.334 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:20.849228+0000", 2020-03-11T22:52:06.335 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.849516+0000" 2020-03-11T22:52:06.335 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.335 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.335 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.335 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.335 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.335 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.336 INFO:tasks.workunit.client.0.smithi130.stdout: "pg": "3.0", 2020-03-11T22:52:06.336 INFO:tasks.workunit.client.0.smithi130.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.336 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [ 2020-03-11T22:52:06.336 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.336 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "40", 2020-03-11T22:52:06.336 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.336 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.337 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.337 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.420592+0000", 2020-03-11T22:52:06.337 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:41.420602+0000" 2020-03-11T22:52:06.337 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.337 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.337 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.337 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.420608+0000", 2020-03-11T22:52:06.338 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:41.527879+0000" 2020-03-11T22:52:06.338 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.338 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.338 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.338 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.527879+0000", 2020-03-11T22:52:06.338 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:41.527942+0000" 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.527943+0000", 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:41.527956+0000" 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.339 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.527956+0000", 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:41.636963+0000" 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.420605+0000", 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:41.636972+0000" 2020-03-11T22:52:06.340 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.341 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.341 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.341 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.637178+0000", 2020-03-11T22:52:06.341 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.596384+0000" 2020-03-11T22:52:06.341 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.341 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.341 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.596384+0000", 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.596436+0000" 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.596436+0000", 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.822253+0000" 2020-03-11T22:52:06.342 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.343 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.343 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.343 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.636972+0000", 2020-03-11T22:52:06.343 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.822278+0000" 2020-03-11T22:52:06.343 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.343 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.343 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.420603+0000", 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.822301+0000" 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:41.420588+0000", 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.822310+0000" 2020-03-11T22:52:06.344 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.345 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.345 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.345 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.345 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "40", 2020-03-11T22:52:06.345 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.345 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.345 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.822310+0000", 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.822494+0000" 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "90", 2020-03-11T22:52:06.346 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.347 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.347 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.347 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.822498+0000", 2020-03-11T22:52:06.347 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.822508+0000" 2020-03-11T22:52:06.347 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.347 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.347 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.822515+0000", 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.842736+0000" 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.842737+0000", 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.842801+0000" 2020-03-11T22:52:06.348 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.842802+0000", 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:42.842815+0000" 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.349 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.842815+0000", 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:43.070359+0000" 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.822512+0000", 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:43.070369+0000" 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.350 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:43.070532+0000", 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:43.613632+0000" 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:43.613633+0000", 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:43.613684+0000" 2020-03-11T22:52:06.351 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:43.613684+0000", 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.635817+0000" 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.352 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:43.070370+0000", 2020-03-11T22:52:06.353 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.635835+0000" 2020-03-11T22:52:06.353 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.353 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.353 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.353 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.822509+0000", 2020-03-11T22:52:06.353 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.635848+0000" 2020-03-11T22:52:06.353 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.354 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.354 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.354 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:42.822495+0000", 2020-03-11T22:52:06.354 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.635851+0000" 2020-03-11T22:52:06.354 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.354 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.354 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "90", 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.635852+0000", 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.636023+0000" 2020-03-11T22:52:06.355 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "106", 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.356 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.636028+0000", 2020-03-11T22:52:06.357 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.636040+0000" 2020-03-11T22:52:06.357 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.357 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.357 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.357 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.636047+0000", 2020-03-11T22:52:06.357 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.670536+0000" 2020-03-11T22:52:06.357 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.670536+0000", 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.670583+0000" 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.358 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.670584+0000", 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:16.670594+0000" 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.670594+0000", 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.045492+0000" 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.359 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.360 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.360 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.636043+0000", 2020-03-11T22:52:06.360 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.045502+0000" 2020-03-11T22:52:06.360 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.360 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.360 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.360 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.045603+0000", 2020-03-11T22:52:06.361 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.119730+0000" 2020-03-11T22:52:06.361 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.361 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.361 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.361 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.119731+0000", 2020-03-11T22:52:06.362 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:49:17.119767+0000" 2020-03-11T22:52:06.362 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.362 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.362 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.362 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.119767+0000", 2020-03-11T22:52:06.362 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.598208+0000" 2020-03-11T22:52:06.362 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:17.045502+0000", 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.598250+0000" 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.363 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.636040+0000", 2020-03-11T22:52:06.364 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.598322+0000" 2020-03-11T22:52:06.364 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.364 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.364 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.364 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:49:16.636024+0000", 2020-03-11T22:52:06.364 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.598370+0000" 2020-03-11T22:52:06.364 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.365 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.365 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.365 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.366 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "106", 2020-03-11T22:52:06.366 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.366 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.366 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.366 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.598371+0000", 2020-03-11T22:52:06.366 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.598835+0000" 2020-03-11T22:52:06.367 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.367 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.367 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.367 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.367 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "108", 2020-03-11T22:52:06.367 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.367 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.598843+0000", 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.598868+0000" 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.598882+0000", 2020-03-11T22:52:06.368 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.606317+0000" 2020-03-11T22:52:06.369 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.369 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.369 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.369 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.606318+0000", 2020-03-11T22:52:06.369 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.606396+0000" 2020-03-11T22:52:06.369 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.369 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.370 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.370 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.598876+0000", 2020-03-11T22:52:06.370 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.606405+0000" 2020-03-11T22:52:06.370 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.370 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.370 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:06.370 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.606406+0000", 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.824702+0000" 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.598868+0000", 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.824740+0000" 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.371 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.372 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.372 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.598835+0000", 2020-03-11T22:52:06.372 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.824744+0000" 2020-03-11T22:52:06.372 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.372 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.372 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.372 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.373 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "108", 2020-03-11T22:52:06.373 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.373 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.373 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.373 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.824745+0000", 2020-03-11T22:52:06.373 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.824993+0000" 2020-03-11T22:52:06.373 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.374 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.374 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.374 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.374 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "116", 2020-03-11T22:52:06.374 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.374 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.374 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.375 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.824999+0000", 2020-03-11T22:52:06.375 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.825015+0000" 2020-03-11T22:52:06.375 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.375 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.376 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.376 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.825025+0000", 2020-03-11T22:52:06.376 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.933612+0000" 2020-03-11T22:52:06.376 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.376 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.377 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.377 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.933613+0000", 2020-03-11T22:52:06.377 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.933694+0000" 2020-03-11T22:52:06.377 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.377 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.377 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.377 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.933695+0000", 2020-03-11T22:52:06.378 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.933715+0000" 2020-03-11T22:52:06.378 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.378 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.378 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.378 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.933715+0000", 2020-03-11T22:52:06.378 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.998433+0000" 2020-03-11T22:52:06.378 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.379 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.379 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.379 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.825020+0000", 2020-03-11T22:52:06.379 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:31.998447+0000" 2020-03-11T22:52:06.379 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.379 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.379 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.380 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.998730+0000", 2020-03-11T22:52:06.380 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:32.061911+0000" 2020-03-11T22:52:06.380 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.380 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.380 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:06.380 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:32.061912+0000", 2020-03-11T22:52:06.380 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:43.923680+0000" 2020-03-11T22:52:06.381 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.381 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.381 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:06.381 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:43.923680+0000", 2020-03-11T22:52:06.381 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:43.924155+0000" 2020-03-11T22:52:06.381 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.381 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.382 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:06.382 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:43.924156+0000", 2020-03-11T22:52:06.382 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:55.938502+0000" 2020-03-11T22:52:06.382 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.382 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.382 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:52:06.382 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:55.938503+0000", 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:55.938728+0000" 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:55.938728+0000", 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.701301+0000" 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.383 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.384 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:06.384 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:20.701301+0000", 2020-03-11T22:52:06.384 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:20.701744+0000" 2020-03-11T22:52:06.384 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.384 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.384 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:06.384 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:20.701744+0000", 2020-03-11T22:52:06.385 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.437279+0000" 2020-03-11T22:52:06.385 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.385 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.385 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.385 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:40.437279+0000", 2020-03-11T22:52:06.385 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.437408+0000" 2020-03-11T22:52:06.385 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.386 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.386 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.386 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:40.437408+0000", 2020-03-11T22:52:06.386 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.629844+0000" 2020-03-11T22:52:06.386 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.386 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.387 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.387 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.998448+0000", 2020-03-11T22:52:06.387 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.629888+0000" 2020-03-11T22:52:06.387 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.387 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.387 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.388 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.825016+0000", 2020-03-11T22:52:06.388 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.629923+0000" 2020-03-11T22:52:06.388 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.388 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.388 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:31.824994+0000", 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.629934+0000" 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "116", 2020-03-11T22:52:06.389 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.390 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.390 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.390 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:40.629935+0000", 2020-03-11T22:52:06.390 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.630467+0000" 2020-03-11T22:52:06.390 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.390 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.390 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: "pg": "1.0", 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [ 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.391 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "27", 2020-03-11T22:52:06.392 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.392 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.392 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Initial", 2020-03-11T22:52:06.392 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.445737+0000", 2020-03-11T22:52:06.392 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.445928+0000" 2020-03-11T22:52:06.392 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.392 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.393 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.393 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.393 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "27", 2020-03-11T22:52:06.393 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.393 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.393 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.393 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.445930+0000", 2020-03-11T22:52:06.394 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.445989+0000" 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "29", 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.445996+0000", 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.446016+0000" 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.446029+0000", 2020-03-11T22:52:06.396 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.467672+0000" 2020-03-11T22:52:06.397 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.397 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.397 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.397 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.467672+0000", 2020-03-11T22:52:06.397 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.467752+0000" 2020-03-11T22:52:06.397 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.397 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.398 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.398 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.467753+0000", 2020-03-11T22:52:06.398 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.467772+0000" 2020-03-11T22:52:06.398 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.398 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.398 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.398 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.467772+0000", 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.595275+0000" 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.446023+0000", 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:37.595309+0000" 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.399 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.400 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.400 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.595539+0000", 2020-03-11T22:52:06.400 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.386652+0000" 2020-03-11T22:52:06.400 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.400 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.400 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.400 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.595310+0000", 2020-03-11T22:52:06.401 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.386678+0000" 2020-03-11T22:52:06.401 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.401 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.401 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.401 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.446017+0000", 2020-03-11T22:52:06.401 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.386694+0000" 2020-03-11T22:52:06.401 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.402 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.402 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.402 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:37.445989+0000", 2020-03-11T22:52:06.402 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.386701+0000" 2020-03-11T22:52:06.402 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.402 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.402 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.403 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.403 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "30", 2020-03-11T22:52:06.403 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.403 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.403 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.403 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.386701+0000", 2020-03-11T22:52:06.403 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.386932+0000" 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "103", 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.404 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.405 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.386935+0000", 2020-03-11T22:52:06.405 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.386947+0000" 2020-03-11T22:52:06.405 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.405 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.405 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.405 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.386955+0000", 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.404762+0000" 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.404763+0000", 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.404827+0000" 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.406 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.407 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.407 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.404827+0000", 2020-03-11T22:52:06.407 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.404840+0000" 2020-03-11T22:52:06.407 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.407 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.407 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.407 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.404841+0000", 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.662761+0000" 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.386952+0000", 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.662772+0000" 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.408 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.409 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.409 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.662898+0000", 2020-03-11T22:52:06.409 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.864198+0000" 2020-03-11T22:52:06.409 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.409 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.409 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.864199+0000", 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:48:38.864249+0000" 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.864249+0000", 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.607230+0000" 2020-03-11T22:52:06.410 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.411 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.411 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.411 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.662773+0000", 2020-03-11T22:52:06.411 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.607264+0000" 2020-03-11T22:52:06.411 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.411 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.411 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.386948+0000", 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.607319+0000" 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:48:38.386932+0000", 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.607327+0000" 2020-03-11T22:52:06.412 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.413 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.413 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.413 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.413 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "103", 2020-03-11T22:52:06.413 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.413 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.413 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.607327+0000", 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.607640+0000" 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "104", 2020-03-11T22:52:06.414 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.607646+0000", 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.607659+0000" 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.415 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.607668+0000", 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.628806+0000" 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.628807+0000", 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.628909+0000" 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.416 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.417 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.417 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.607664+0000", 2020-03-11T22:52:06.417 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.628926+0000" 2020-03-11T22:52:06.417 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.417 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.417 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:52:06.417 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.628926+0000", 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.799136+0000" 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.607660+0000", 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.799160+0000" 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.418 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.419 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.419 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.607641+0000", 2020-03-11T22:52:06.419 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.799168+0000" 2020-03-11T22:52:06.419 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.419 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.419 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.419 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.420 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "104", 2020-03-11T22:52:06.420 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.420 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.420 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.420 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.799169+0000", 2020-03-11T22:52:06.420 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.799533+0000" 2020-03-11T22:52:06.420 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "118", 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2020-03-11T22:52:06.421 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.799542+0000", 2020-03-11T22:52:06.422 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.799567+0000" 2020-03-11T22:52:06.422 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.422 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.422 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:52:06.422 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.799583+0000", 2020-03-11T22:52:06.422 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.820554+0000" 2020-03-11T22:52:06.422 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.820554+0000", 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.820685+0000" 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:52:06.423 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.820685+0000", 2020-03-11T22:52:06.424 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:23.820741+0000" 2020-03-11T22:52:06.424 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.424 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.424 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:52:06.424 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.820742+0000", 2020-03-11T22:52:06.424 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:24.076072+0000" 2020-03-11T22:52:06.424 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.425 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.425 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:52:06.425 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.799576+0000", 2020-03-11T22:52:06.425 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:24.076085+0000" 2020-03-11T22:52:06.425 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.425 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.425 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:52:06.426 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:24.076396+0000", 2020-03-11T22:52:06.426 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:50:24.585788+0000" 2020-03-11T22:52:06.426 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.426 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.426 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:52:06.426 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:24.585789+0000", 2020-03-11T22:52:06.426 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.437592+0000" 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:40.437592+0000", 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:51:40.437958+0000" 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.427 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:51:40.437958+0000", 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:52:01.339764+0000" 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:52:01.339764+0000", 2020-03-11T22:52:06.428 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:52:01.339932+0000" 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:52:01.339932+0000", 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:52:01.614939+0000" 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.429 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.430 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2020-03-11T22:52:06.430 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:24.076085+0000", 2020-03-11T22:52:06.430 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:52:01.614976+0000" 2020-03-11T22:52:06.430 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.430 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.430 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2020-03-11T22:52:06.430 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.799568+0000", 2020-03-11T22:52:06.431 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:52:01.615010+0000" 2020-03-11T22:52:06.431 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.431 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.431 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2020-03-11T22:52:06.431 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:50:23.799533+0000", 2020-03-11T22:52:06.431 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:52:01.615021+0000" 2020-03-11T22:52:06.431 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.432 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.432 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:52:06.432 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.432 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "118", 2020-03-11T22:52:06.432 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2020-03-11T22:52:06.432 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:52:06.432 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2020-03-11T22:52:01.615022+0000", 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2020-03-11T22:52:01.615498+0000" 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:52:06.433 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:52:06.434 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:52:06.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:52:06.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-03-11T22:52:06.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-03-11T22:52:06.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:52:06.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:52:06.435 INFO:tasks.workunit.client.0.smithi130.stdout:TEST PASSED 2020-03-11T22:52:06.789 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' does not exist 2020-03-11T22:52:06.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-03-11T22:52:06.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:52:06.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:52:07.264 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' does not exist 2020-03-11T22:52:07.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-03-11T22:52:07.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-03-11T22:52:07.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-03-11T22:52:07.805 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test7' does not exist 2020-03-11T22:52:07.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:52:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:07.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:07.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:07.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:07.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:07.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-03-11T22:52:07.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:52:07.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:52:07.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:07.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:52:07.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:07.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:07.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:07.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:07.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:07.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:07.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:07.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:07.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:07.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:07.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:07.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:07.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:07.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:07.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:07.933 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:07.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:52:07.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:07.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:07.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:52:07.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56067 2020-03-11T22:52:07.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:07.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:07.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:52:07.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/osd-backfill-prio 0 2020-03-11T22:52:07.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:52:07.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:52:07.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:52:07.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:07.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:07.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:07.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:07.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:07.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:07.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:07.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:07.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:07.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:07.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:07.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:07.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:07.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:07.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:07.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:07.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:07.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:52:07.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:52:07.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:07.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:07.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56067 2020-03-11T22:52:07.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56067 2020-03-11T22:52:07.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:07.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:07.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:52:07.973 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:52:07.973 INFO:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:52:08.048 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-03-11T22:52:08.048 INFO:teuthology.orchestra.run.smithi130:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-03-11T22:52:08.106 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:52:08.114 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:52:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:52:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:52:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:52:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:52:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:52:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:52:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:52:08.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:52:08.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-recovery-log 2020-03-11T22:52:08.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-03-11T22:52:08.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-03-11T22:52:08.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-03-11T22:52:08.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-03-11T22:52:08.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-03-11T22:52:08.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none ' 2020-03-11T22:52:08.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:08.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-03-11T22:52:08.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:52:08.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-03-11T22:52:08.119 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_log_2 2020-03-11T22:52:08.120 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_recovery_1 2020-03-11T22:52:08.120 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_recovery_2' 2020-03-11T22:52:08.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:52:08.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:52:08.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:52:08.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:08.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:52:08.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:08.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:08.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:08.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:08.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:08.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:08.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:08.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:08.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:08.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:08.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:08.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:08.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:08.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:08.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:52:08.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:08.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:08.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:08.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76578 2020-03-11T22:52:08.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:08.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:08.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:52:08.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:52:08.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:08.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76578 2020-03-11T22:52:08.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:52:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:52:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:52:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:52:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:52:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-03-11T22:52:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:103: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-03-11T22:52:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:52:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:52:08.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:52:08.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:52:08.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:52:08.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:08.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:08.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:52:08.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:52:08.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:52:08.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:08.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:08.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:52:08.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:52:08.601 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:08.601 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:08.601 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:08.601 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:08.601 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:08.601 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:08.602 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76578/ceph-mon.a.asok 2020-03-11T22:52:08.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:08.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76578/ceph-mon.a.asok config get fsid 2020-03-11T22:52:08.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:52:08.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:08.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:08.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:52:08.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:52:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76578/ceph-mon.a.asok 2020-03-11T22:52:08.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:08.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76578/ceph-mon.a.asok config get mon_host 2020-03-11T22:52:08.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:52:08.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:08.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:52:08.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:52:08.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:52:08.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:52:08.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:52:09.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:52:09.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:09.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:09.648 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:09.649 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:09.649 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:09.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:09.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:09.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:52:09.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:09.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:09.672 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:52:09.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:52:09.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:09.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:52:09.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:09.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:09.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:52:09.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:09.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:52:09.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:09.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:09.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:09.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:09.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:09.679 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:09.679 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:09.679 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:09.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:09.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:09.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:09.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:09.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:09.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:09.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:09.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:09.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:09.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:09.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:09.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:09.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:52:09.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:09.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d8493f62-9ffa-42c3-92b5-dfac4db73b5e 2020-03-11T22:52:09.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 d8493f62-9ffa-42c3-92b5-dfac4db73b5e' 2020-03-11T22:52:09.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:09.682 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 d8493f62-9ffa-42c3-92b5-dfac4db73b5e 2020-03-11T22:52:09.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCZa2letljzKBAA5vM5I8eR2nLjt2vQQuwa6g== 2020-03-11T22:52:09.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCZa2letljzKBAA5vM5I8eR2nLjt2vQQuwa6g=="}' 2020-03-11T22:52:09.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d8493f62-9ffa-42c3-92b5-dfac4db73b5e -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:10.064 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:52:10.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:10.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCZa2letljzKBAA5vM5I8eR2nLjt2vQQuwa6g== --osd-uuid d8493f62-9ffa-42c3-92b5-dfac4db73b5e 2020-03-11T22:52:10.097 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:10.095+0000 7f3fd6466ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:10.099 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:10.097+0000 7f3fd6466ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:10.101 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:10.099+0000 7f3fd6466ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:10.099+0000 7f3fd6466ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:10.099+0000 7f3fd6466ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:10.102 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:10.099+0000 7f3fd6466ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:10.147 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:10.145+0000 7f3fd6466ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:52:13.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:52:13.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:13.100 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:52:13.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:52:13.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:13.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:52:13.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:13.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:13.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:13.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:13.563 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:52:13.584 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:13.581+0000 7f783b896ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:13.585 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:13.583+0000 7f783b896ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:13.587 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:13.585+0000 7f783b896ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:13.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:52:13.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:13.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:52:13.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:13.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:13.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:13.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:13.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:13.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:13.932 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:52:14.103 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:14.100+0000 7f783b896ec0 -1 Falling back to public interface 2020-03-11T22:52:14.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:14.871 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:14.868+0000 7f783b896ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:52:15.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:15.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:15.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:15.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:15.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:15.275 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:52:15.689 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77018,v1:127.0.0.1:6803/77018] [v2:127.0.0.1:6804/77018,v1:127.0.0.1:6805/77018] exists,up d8493f62-9ffa-42c3-92b5-dfac4db73b5e 2020-03-11T22:52:15.689 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 09ad104b-9c81-46b5-9dcf-56e4dd4407bb 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:15.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:15.693 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:15.693 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:15.693 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:15.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:15.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:15.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:15.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:15.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:52:15.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:15.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=09ad104b-9c81-46b5-9dcf-56e4dd4407bb 2020-03-11T22:52:15.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 09ad104b-9c81-46b5-9dcf-56e4dd4407bb' 2020-03-11T22:52:15.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:15.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCfa2le7b6fKRAACY3mVolaxiBIJsJzvfxH+A== 2020-03-11T22:52:15.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCfa2le7b6fKRAACY3mVolaxiBIJsJzvfxH+A=="}' 2020-03-11T22:52:15.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 09ad104b-9c81-46b5-9dcf-56e4dd4407bb -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:16.226 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:52:16.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:16.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCfa2le7b6fKRAACY3mVolaxiBIJsJzvfxH+A== --osd-uuid 09ad104b-9c81-46b5-9dcf-56e4dd4407bb 2020-03-11T22:52:16.260 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:16.257+0000 7f5f01c3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:16.262 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:16.259+0000 7f5f01c3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:16.262 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:16.260+0000 7f5f01c3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:16.263 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:16.260+0000 7f5f01c3eec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:16.263 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:16.260+0000 7f5f01c3eec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:16.263 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:16.260+0000 7f5f01c3eec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:16.335 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:16.332+0000 7f5f01c3eec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:52:19.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:52:19.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:19.312 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:52:19.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:52:19.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:19.769 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:52:19.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:52:19.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:19.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:19.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:19.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:19.791 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:19.789+0000 7f3d49ee3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:19.793 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:19.791+0000 7f3d49ee3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:19.794 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:19.792+0000 7f3d49ee3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:20.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:52:20.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:20.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:52:20.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:20.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:20.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:20.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:20.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:20.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:20.115 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:52:20.310 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:20.307+0000 7f3d49ee3ec0 -1 Falling back to public interface 2020-03-11T22:52:20.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:21.074 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:21.070+0000 7f3d49ee3ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:52:21.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:21.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:21.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:21.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:21.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:21.454 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:52:21.828 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/77373,v1:127.0.0.1:6811/77373] [v2:127.0.0.1:6812/77373,v1:127.0.0.1:6813/77373] exists,up 09ad104b-9c81-46b5-9dcf-56e4dd4407bb 2020-03-11T22:52:21.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:21.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:21.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:21.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:21.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:52:21.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:21.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:21.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:52:21.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:21.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:52:21.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:21.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:21.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:21.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:21.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:21.831 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:21.831 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:21.831 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:21.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:52:21.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:21.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=438c8f60-15de-44cf-9aba-54f32c31b4c7 2020-03-11T22:52:21.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 438c8f60-15de-44cf-9aba-54f32c31b4c7' 2020-03-11T22:52:21.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:21.835 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 438c8f60-15de-44cf-9aba-54f32c31b4c7 2020-03-11T22:52:21.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCla2lelMIWMhAAqxPDKKA4/0JE+xiMfccxoA== 2020-03-11T22:52:21.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCla2lelMIWMhAAqxPDKKA4/0JE+xiMfccxoA=="}' 2020-03-11T22:52:21.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 438c8f60-15de-44cf-9aba-54f32c31b4c7 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:22.308 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:52:22.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:22.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCla2lelMIWMhAAqxPDKKA4/0JE+xiMfccxoA== --osd-uuid 438c8f60-15de-44cf-9aba-54f32c31b4c7 2020-03-11T22:52:22.342 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:22.339+0000 7f92e0048ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:22.344 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:22.340+0000 7f92e0048ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:22.345 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:22.341+0000 7f92e0048ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:22.345 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:22.342+0000 7f92e0048ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:22.346 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:22.342+0000 7f92e0048ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:22.346 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:22.342+0000 7f92e0048ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:22.431 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:22.427+0000 7f92e0048ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:52:25.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:52:25.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:25.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:52:25.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:25.424 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:52:25.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:52:25.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:25.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:25.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:25.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:25.875 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:52:25.895 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:25.892+0000 7feb40509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:25.897 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:25.894+0000 7feb40509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:25.899 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:25.895+0000 7feb40509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:26.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:52:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:52:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:26.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:26.233 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:52:26.414 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:26.410+0000 7feb40509ec0 -1 Falling back to public interface 2020-03-11T22:52:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:27.187 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:27.183+0000 7feb40509ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:52:27.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:27.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:27.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:27.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:27.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:27.576 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:52:27.931 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/77728,v1:127.0.0.1:6819/77728] [v2:127.0.0.1:6820/77728,v1:127.0.0.1:6821/77728] exists,up 438c8f60-15de-44cf-9aba-54f32c31b4c7 2020-03-11T22:52:27.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:27.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:27.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:27.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:27.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:27.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:27.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:27.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:52:27.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:27.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b674b434-3ea7-43cb-85cd-b1f7c3d99ae8 2020-03-11T22:52:27.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 b674b434-3ea7-43cb-85cd-b1f7c3d99ae8' 2020-03-11T22:52:27.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:27.938 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 b674b434-3ea7-43cb-85cd-b1f7c3d99ae8 2020-03-11T22:52:27.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCra2leRq5COBAAxx4202eWu/PKd1UPJEo0AQ== 2020-03-11T22:52:27.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCra2leRq5COBAAxx4202eWu/PKd1UPJEo0AQ=="}' 2020-03-11T22:52:27.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b674b434-3ea7-43cb-85cd-b1f7c3d99ae8 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:28.385 INFO:tasks.workunit.client.0.smithi130.stdout:3 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:28.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCra2leRq5COBAAxx4202eWu/PKd1UPJEo0AQ== --osd-uuid b674b434-3ea7-43cb-85cd-b1f7c3d99ae8 2020-03-11T22:52:28.418 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:28.415+0000 7f68eded6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:28.441+0000 7f68eded6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:28.446 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:28.443+0000 7f68eded6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:28.443+0000 7f68eded6ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:28.443+0000 7f68eded6ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:28.447 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:28.443+0000 7f68eded6ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:28.553 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:28.549+0000 7f68eded6ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:52:31.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:52:31.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:31.580 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2020-03-11T22:52:31.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:52:31.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:32.034 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2020-03-11T22:52:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:52:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:32.056 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:32.052+0000 7f955882fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:32.057 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:32.054+0000 7f955882fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:32.059 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:32.056+0000 7f955882fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:32.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:52:32.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:32.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:52:32.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:32.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:32.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:32.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:32.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:32.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:32.403 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:52:32.575 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:32.571+0000 7f955882fec0 -1 Falling back to public interface 2020-03-11T22:52:32.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:33.326 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:33.322+0000 7f955882fec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:52:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:33.749 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:52:34.128 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78083,v1:127.0.0.1:6827/78083] [v2:127.0.0.1:6828/78083,v1:127.0.0.1:6829/78083] exists,up b674b434-3ea7-43cb-85cd-b1f7c3d99ae8 2020-03-11T22:52:34.128 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 e42ea571-f03f-49f8-9cf6-05366eb4cd61 2020-03-11T22:52:34.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:34.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:34.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:34.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:34.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:52:34.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:34.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:34.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:52:34.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:34.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:52:34.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:34.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:34.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:34.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:34.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:52:34.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:52:34.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:34.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:34.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:34.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:34.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:34.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:34.132 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:34.132 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:34.132 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:34.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:34.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:34.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:34.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:34.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:34.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:34.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:34.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:34.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:34.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:34.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:34.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:34.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:52:34.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:34.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e42ea571-f03f-49f8-9cf6-05366eb4cd61 2020-03-11T22:52:34.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 e42ea571-f03f-49f8-9cf6-05366eb4cd61' 2020-03-11T22:52:34.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:34.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCya2leSfFSCBAA7H3LYwqubyG8VN8/6TsZ4w== 2020-03-11T22:52:34.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCya2leSfFSCBAA7H3LYwqubyG8VN8/6TsZ4w=="}' 2020-03-11T22:52:34.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e42ea571-f03f-49f8-9cf6-05366eb4cd61 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:52:34.580 INFO:tasks.workunit.client.0.smithi130.stdout:4 2020-03-11T22:52:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:52:34.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCya2leSfFSCBAA7H3LYwqubyG8VN8/6TsZ4w== --osd-uuid e42ea571-f03f-49f8-9cf6-05366eb4cd61 2020-03-11T22:52:34.612 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:34.609+0000 7fba88b16ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:34.614 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:34.611+0000 7fba88b16ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:34.616 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:34.612+0000 7fba88b16ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:34.616 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:34.612+0000 7fba88b16ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:34.617 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:34.613+0000 7fba88b16ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:34.617 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:34.613+0000 7fba88b16ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:52:34.690 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:34.687+0000 7fba88b16ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:52:37.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:52:37.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:37.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:52:37.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:37.780 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2020-03-11T22:52:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:52:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:38.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:38.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:38.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:38.234 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2020-03-11T22:52:38.255 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:38.252+0000 7f3a20179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:38.257 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:38.253+0000 7f3a20179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:38.258 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:38.255+0000 7f3a20179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:38.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:52:38.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:38.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:52:38.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:38.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:38.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:38.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:38.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:38.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:52:38.598 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:52:38.780 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:38.776+0000 7f3a20179ec0 -1 Falling back to public interface 2020-03-11T22:52:38.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:39.540 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:39.536+0000 7f3a20179ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:52:39.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:39.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:39.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:39.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:39.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:52:39.971 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:52:40.376 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/78438,v1:127.0.0.1:6835/78438] [v2:127.0.0.1:6836/78438,v1:127.0.0.1:6837/78438] exists,up e42ea571-f03f-49f8-9cf6-05366eb4cd61 2020-03-11T22:52:40.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:40.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:40.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:40.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:40.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:52:40.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:40.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:40.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:52:40.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:40.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:52:40.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:40.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:40.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:40.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:40.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:52:40.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:52:40.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:40.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:52:40.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:40.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:52:40.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:40.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:40.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:40.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:40.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:40.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:40.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:40.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:40.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:40.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:40.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:52:40.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:40.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=326cb1d0-a60b-4c0b-a9b8-8e4f4f029379 2020-03-11T22:52:40.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 326cb1d0-a60b-4c0b-a9b8-8e4f4f029379' 2020-03-11T22:52:40.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:40.382 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 326cb1d0-a60b-4c0b-a9b8-8e4f4f029379 2020-03-11T22:52:40.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC4a2leBHEiFxAAr4xwjNwSFh5CjfbfVMlLJw== 2020-03-11T22:52:40.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC4a2leBHEiFxAAr4xwjNwSFh5CjfbfVMlLJw=="}' 2020-03-11T22:52:40.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 326cb1d0-a60b-4c0b-a9b8-8e4f4f029379 -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:52:40.895 INFO:tasks.workunit.client.0.smithi130.stdout:5 2020-03-11T22:52:40.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:52:40.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC4a2leBHEiFxAAr4xwjNwSFh5CjfbfVMlLJw== --osd-uuid 326cb1d0-a60b-4c0b-a9b8-8e4f4f029379 2020-03-11T22:52:40.930 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:40.926+0000 7f01b6cb4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.931 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:40.928+0000 7f01b6cb4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.933 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:40.930+0000 7f01b6cb4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.934 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:40.930+0000 7f01b6cb4ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:52:40.934 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:40.930+0000 7f01b6cb4ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:52:40.934 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:40.930+0000 7f01b6cb4ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:52:40.997 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:40.994+0000 7f01b6cb4ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:52:44.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:52:44.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:44.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:52:44.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:44.141 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2020-03-11T22:52:44.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:52:44.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:44.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:44.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:44.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:44.600 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2020-03-11T22:52:44.622 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:44.618+0000 7f0e54c34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:44.623 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:44.620+0000 7f0e54c34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:44.624 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:44.621+0000 7f0e54c34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:44.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:52:44.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:44.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:52:44.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:44.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:44.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:44.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:44.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:52:44.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:44.942 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:52:45.149 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:45.145+0000 7f0e54c34ec0 -1 Falling back to public interface 2020-03-11T22:52:45.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:45.914 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:52:45.910+0000 7f0e54c34ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:52:46.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:46.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:46.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:46.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:52:46.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:46.311 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:52:46.706 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/78793,v1:127.0.0.1:6843/78793] [v2:127.0.0.1:6844/78793,v1:127.0.0.1:6845/78793] exists,up 326cb1d0-a60b-4c0b-a9b8-8e4f4f029379 2020-03-11T22:52:46.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:46.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:46.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:46.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:52:46.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:52:47.232 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2020-03-11T22:52:47.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:48.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:52:48.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:52:48.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:52:48.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:52:48.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:52:48.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:52:48.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:52:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:52:48.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:52:48.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:48.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:52:49.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:52:49.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:52:49.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:52:49.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:52:49.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:52:49.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:52:49.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:52:49.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:52:49.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:52:49.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:52:49.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:52:49.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:52:49.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:52:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:52:50.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:52:50.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:52:50.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:52:50.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:52:50.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:52:50.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:52:50.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:52:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:52:50.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:52:50.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:52:50.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:52:50.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:52:50.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:52:50.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:52:50.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:50.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:52:51.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:52:51.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:52:51.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:52:51.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:52:51.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:52:51.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:52:51.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:52:51.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:52:51.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:52:51.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:52:51.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:52:51.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:52:51.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:52:51.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:51.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:52:52.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:52:52.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:52:52.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:52:52.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:52:52.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:52:52.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:52:52.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:52:52.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:52:52.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:52:52.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:52:52.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:52:52.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:52:52.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:52:52.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:52:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:52:52.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:52:52.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:52.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:52:53.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:52:53.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:52:53.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:52:53.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:52:53.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:52:53.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:52:53.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:52:53.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:52:53.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:52:53.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:52:53.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:52:53.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:52:53.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:52:53.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:53.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:52:54.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:52:54.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:52:54.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:52:54.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:52:54.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:52:54.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:52:54.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:52:54.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:52:54.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:52:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:52:54.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:52:54.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:52:54.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:52:54.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:52:54.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:54.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:52:55.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:52:55.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:52:55.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:52:55.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:52:55.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:52:55.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:52:55.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:52:55.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:52:55.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:52:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:52:55.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:52:55.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:52:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:52:55.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:52:55.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:55.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:52:56.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:52:56.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:52:56.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:52:56.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:52:56.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:52:56.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:52:56.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:52:56.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:52:56.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:52:56.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:52:56.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:52:56.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:52:56.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:52:56.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:56.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:52:57.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:52:57.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:52:57.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:52:57.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:52:57.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:52:57.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:52:57.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:52:57.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:52:57.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:52:57.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:52:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:52:57.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:52:57.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:52:57.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:57.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:52:58.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:52:58.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:52:58.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:52:58.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:52:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:52:58.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:52:58.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:52:58.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:52:58.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:58.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:52:58.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:52:58.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:52:58.973 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:52:58.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:52:59.465 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:52:59.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:52:59.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:53:00.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:53:00.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:53:00.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:53:00.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:53:00.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:53:00.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:53:00.476 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:53:00.476 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:53:00.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:53:00.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:53:00.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:53:00.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:53:00.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:53:00.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:53:00.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:00.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:00.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:00.917 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:53:00.917 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:53:00.917 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:53:00.917 INFO:tasks.workunit.client.0.smithi130.stderr:4 2020-03-11T22:53:00.917 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2020-03-11T22:53:00.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:00.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:00.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:01.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836491 2020-03-11T22:53:01.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491' 2020-03-11T22:53:01.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:01.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:01.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:53:01.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672971' 2020-03-11T22:53:01.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:01.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:01.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542154 2020-03-11T22:53:01.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-60129542154' 2020-03-11T22:53:01.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:01.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:01.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=77309411337 2020-03-11T22:53:01.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-60129542154 3-77309411337' 2020-03-11T22:53:01.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:01.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:01.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=94489280519 2020-03-11T22:53:01.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-60129542154 3-77309411337 4-94489280519' 2020-03-11T22:53:01.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:01.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:53:01.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=115964116997 2020-03-11T22:53:01.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-60129542154 3-77309411337 4-94489280519 5-115964116997' 2020-03-11T22:53:01.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:01.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836491 2020-03-11T22:53:01.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:01.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:01.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836491 2020-03-11T22:53:01.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:01.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836491 2020-03-11T22:53:01.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-03-11T22:53:01.668 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836491 2020-03-11T22:53:01.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:02.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836491 2020-03-11T22:53:02.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:03.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:03.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:03.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836491 2020-03-11T22:53:03.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:03.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:03.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:03.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:03.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:03.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:03.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:53:03.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:53:03.367 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:53:03.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:03.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672971 2020-03-11T22:53:03.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:03.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542154 2020-03-11T22:53:03.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:03.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:03.731 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542154 2020-03-11T22:53:03.731 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:03.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542154 2020-03-11T22:53:03.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2020-03-11T22:53:03.733 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 60129542154 2020-03-11T22:53:03.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:04.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542154 -lt 60129542154 2020-03-11T22:53:04.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:04.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-77309411337 2020-03-11T22:53:04.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:04.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:04.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-77309411337 2020-03-11T22:53:04.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:04.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=77309411337 2020-03-11T22:53:04.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 77309411337' 2020-03-11T22:53:04.082 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 77309411337 2020-03-11T22:53:04.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:04.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 77309411337 -lt 77309411337 2020-03-11T22:53:04.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:04.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-94489280519 2020-03-11T22:53:04.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:04.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-94489280519 2020-03-11T22:53:04.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:04.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=94489280519 2020-03-11T22:53:04.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 94489280519' 2020-03-11T22:53:04.433 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 94489280519 2020-03-11T22:53:04.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:04.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 94489280519 -lt 94489280519 2020-03-11T22:53:04.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:04.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-115964116997 2020-03-11T22:53:04.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:04.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:53:04.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-115964116997 2020-03-11T22:53:04.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:04.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=115964116997 2020-03-11T22:53:04.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 115964116997' 2020-03-11T22:53:04.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:53:04.783 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 115964116997 2020-03-11T22:53:05.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 115964116998 -lt 115964116997 2020-03-11T22:53:05.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:53:05.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:05.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:05.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:53:05.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:05.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:05.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:05.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:05.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:05.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:05.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:05.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:05.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:05.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:06.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:06.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:53:06.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:53:06.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:53:06.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:06.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:06.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:06.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:06.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:06.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:06.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:06.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:06.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:06.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:06.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:06.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:06.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:06.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:07.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:07.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:07.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:07.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:07.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:07.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:07.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:07.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:07.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:07.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25495 2020-03-11T22:53:07.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25495 '!=' null 2020-03-11T22:53:07.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:07.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:07.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:07.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:07.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:07.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:08.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:08.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:08.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:08.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:08.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:08.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:08.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:08.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:08.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:08.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:08.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:08.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:08.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:09.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20693 2020-03-11T22:53:09.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20693 '!=' null 2020-03-11T22:53:09.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:09.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:09.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:09.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:09.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:09.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:09.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:09.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:09.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:09.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:09.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:09.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:09.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:09.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:09.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:10.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:10.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:10.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:10.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:10.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:10.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:10.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:10.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:10.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:10.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20693 2020-03-11T22:53:10.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20693 '!=' null 2020-03-11T22:53:10.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:10.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:10.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:10.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:10.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:10.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:10.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:10.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:10.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:10.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:10.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:10.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:10.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:10.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:10.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:11.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:11.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:11.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:11.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:11.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:11.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:11.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17415 2020-03-11T22:53:11.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17415 '!=' null 2020-03-11T22:53:11.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:11.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:11.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:11.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:11.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:11.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:11.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:11.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:11.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:11.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:11.948 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:12.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:12.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:12.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:12.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:12.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:12.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:12.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:13.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17415 2020-03-11T22:53:13.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17415 '!=' null 2020-03-11T22:53:13.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:13.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:13.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:13.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:13.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:13.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:13.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:14.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:14.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:14.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:14.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:14.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:14.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:14.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:14.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:14.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:14.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31623 2020-03-11T22:53:14.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31623 '!=' null 2020-03-11T22:53:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:14.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:14.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:14.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:14.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:14.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:14.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:14.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:14.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:15.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:15.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:15.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:15.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:15.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:15.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:15.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:15.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:15.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:15.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:15.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:15.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:15.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:15.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:53:15.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:53:15.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:15.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:15.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:16.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:16.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:16.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:16.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:16.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:16.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:16.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:16.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:16.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:16.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:16.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:16.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:16.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:16.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:16.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:16.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:16.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:16.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:16.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:16.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:16.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:17.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45998 2020-03-11T22:53:17.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45998 '!=' null 2020-03-11T22:53:17.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:17.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:17.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:17.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:17.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:17.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:17.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:17.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:17.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:17.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:17.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:17.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:17.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:17.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:17.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:18.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:18.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:18.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:18.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:18.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:18.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:18.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:18.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:18.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:18.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45998 2020-03-11T22:53:18.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45998 '!=' null 2020-03-11T22:53:18.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:18.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:18.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:18.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:18.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:18.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:18.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:18.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:18.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:18.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:19.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:19.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:19.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:19.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:19.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:19.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:19.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:19.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:20.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30473 2020-03-11T22:53:20.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30473 '!=' null 2020-03-11T22:53:20.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:20.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:20.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:20.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:20.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:20.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:20.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:20.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:20.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:20.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:20.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:20.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:20.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:20.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:20.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:20.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:20.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:20.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:20.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:21.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30473 2020-03-11T22:53:21.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30473 '!=' null 2020-03-11T22:53:21.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:21.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:21.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:21.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:21.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:21.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:21.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:21.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:21.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:21.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:21.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:21.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:21.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:22.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:22.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:22.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:22.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:22.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:22.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:22.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30473 2020-03-11T22:53:22.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30473 '!=' null 2020-03-11T22:53:22.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:22.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:22.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:22.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:22.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:22.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:22.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:22.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:22.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:22.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:22.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:23.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:23.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:23.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:23.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:23.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:23.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:23.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:23.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:23.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:23.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:23.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:23.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:23.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:24.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=47722 2020-03-11T22:53:24.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 47722 '!=' null 2020-03-11T22:53:24.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:24.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:24.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:24.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:24.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:24.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:24.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:24.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:24.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:24.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:24.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:24.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:24.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:24.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:24.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:24.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:24.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:24.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:24.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:24.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:24.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:24.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:24.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:24.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:25.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32198 2020-03-11T22:53:25.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32198 '!=' null 2020-03-11T22:53:25.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:25.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:25.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:25.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:25.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:25.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:25.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:25.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:25.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:25.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:25.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:25.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:25.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:25.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:25.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:26.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:26.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:26.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:26.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:26.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:26.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:26.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:26.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:26.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:26.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32198 2020-03-11T22:53:26.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32198 '!=' null 2020-03-11T22:53:26.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:26.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:26.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:26.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:26.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:26.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:26.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:26.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:26.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:26.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:26.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:27.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:27.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:27.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:27.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:27.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:27.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:27.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:27.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:27.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:27.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:27.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:27.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:27.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:28.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=48873 2020-03-11T22:53:28.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 48873 '!=' null 2020-03-11T22:53:28.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:28.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:28.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:28.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:28.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:28.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:28.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:28.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:28.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:28.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:28.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:28.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:28.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:28.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:28.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:28.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:28.948 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:28.948 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=50886 2020-03-11T22:53:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 50886 '!=' null 2020-03-11T22:53:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:29.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:29.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:29.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:29.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:29.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:29.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:29.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:29.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:29.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:29.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:29.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:29.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:30.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:30.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:30.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:30.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:30.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:30.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:30.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:30.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:30.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:30.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=50886 2020-03-11T22:53:30.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 50886 '!=' null 2020-03-11T22:53:30.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:30.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:30.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:30.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:30.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:30.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:30.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:30.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:30.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:30.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:30.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:31.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:53:31.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:31.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:31.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:31.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:53:31.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:53:31.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:53:31.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:53:31.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:53:31.958 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:53:31.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:53:31.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:53:31.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:31.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:31.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:31.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:31.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:32.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:32.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:53:32.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:53:32.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:32.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:32.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:53:32.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:53:32.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:53:32.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:53:32.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:53:33.191 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:53:33.191 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_log_t": { 2020-03-11T22:53:33.191 INFO:tasks.workunit.client.0.smithi130.stdout: "head": "30'150", 2020-03-11T22:53:33.192 INFO:tasks.workunit.client.0.smithi130.stdout: "tail": "30'148", 2020-03-11T22:53:33.192 INFO:tasks.workunit.client.0.smithi130.stdout: "log": [ 2020-03-11T22:53:33.192 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.192 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2020-03-11T22:53:33.192 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:53:33.192 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'149", 2020-03-11T22:53:33.192 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2020-03-11T22:53:33.193 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4631.0:1", 2020-03-11T22:53:33.193 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2020-03-11T22:53:33.193 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2020-03-11T22:52:58.530752+0000", 2020-03-11T22:53:33.193 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2020-03-11T22:53:33.193 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2020-03-11T22:53:33.193 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2020-03-11T22:53:33.194 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:53:33.195 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2020-03-11T22:53:33.195 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2020-03-11T22:53:33.195 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.195 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.195 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.195 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.195 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'150", 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4634.0:1", 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2020-03-11T22:52:58.631049+0000", 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2020-03-11T22:53:33.196 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2020-03-11T22:53:33.197 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2020-03-11T22:53:33.197 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2020-03-11T22:53:33.197 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2020-03-11T22:53:33.197 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2020-03-11T22:53:33.197 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.197 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.197 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.198 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:53:33.199 INFO:tasks.workunit.client.0.smithi130.stdout: "dups": [ 2020-03-11T22:53:33.199 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.199 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4607.0:1", 2020-03-11T22:53:33.199 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'141", 2020-03-11T22:53:33.199 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "141", 2020-03-11T22:53:33.199 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.199 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4610.0:1", 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'142", 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "142", 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.200 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4613.0:1", 2020-03-11T22:53:33.201 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'143", 2020-03-11T22:53:33.201 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "143", 2020-03-11T22:53:33.201 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.201 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.201 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.201 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4616.0:1", 2020-03-11T22:53:33.201 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'144", 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "144", 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4619.0:1", 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'145", 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "145", 2020-03-11T22:53:33.202 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4622.0:1", 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'146", 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "146", 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.203 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4625.0:1", 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'147", 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "147", 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4628.0:1", 2020-03-11T22:53:33.204 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'148", 2020-03-11T22:53:33.205 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "148", 2020-03-11T22:53:33.205 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:53:33.205 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.205 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:53:33.205 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:53:33.205 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_missing_t": { 2020-03-11T22:53:33.205 INFO:tasks.workunit.client.0.smithi130.stdout: "missing": [], 2020-03-11T22:53:33.206 INFO:tasks.workunit.client.0.smithi130.stdout: "may_include_deletes": true 2020-03-11T22:53:33.206 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:53:33.206 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:53:33.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:53:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:53:33.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:33.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=8 2020-03-11T22:53:33.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 8 '!=' 8 ']' 2020-03-11T22:53:33.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:53:33.208 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to max 2 2020-03-11T22:53:33.208 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to copy log version 30'150 2020-03-11T22:53:33.208 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to copy log version 30'149 2020-03-11T22:53:33.208 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:33.209 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'141 2020-03-11T22:53:33.209 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'142 2020-03-11T22:53:33.209 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'143 2020-03-11T22:53:33.209 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'144 2020-03-11T22:53:33.209 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'145 2020-03-11T22:53:33.209 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'146 2020-03-11T22:53:33.209 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'147 2020-03-11T22:53:33.210 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.943+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'148 2020-03-11T22:53:33.210 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to max 2 2020-03-11T22:53:33.210 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to copy log version 30'150 2020-03-11T22:53:33.210 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to copy log version 30'149 2020-03-11T22:53:33.210 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:33.210 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'141 2020-03-11T22:53:33.210 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'142 2020-03-11T22:53:33.211 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'143 2020-03-11T22:53:33.211 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'144 2020-03-11T22:53:33.211 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'145 2020-03-11T22:53:33.211 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'146 2020-03-11T22:53:33.211 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'147 2020-03-11T22:53:33.211 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'148 2020-03-11T22:53:33.211 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to max 2 2020-03-11T22:53:33.212 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to copy log version 30'150 2020-03-11T22:53:33.212 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to copy log version 30'149 2020-03-11T22:53:33.212 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:33.212 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'141 2020-03-11T22:53:33.212 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'142 2020-03-11T22:53:33.212 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'143 2020-03-11T22:53:33.219 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'144 2020-03-11T22:53:33.219 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'145 2020-03-11T22:53:33.219 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'146 2020-03-11T22:53:33.219 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'147 2020-03-11T22:53:33.220 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:59.944+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'148 2020-03-11T22:53:33.220 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_after v 30'150 2020-03-11T22:53:33.220 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:53:33.220 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'141 2020-03-11T22:53:33.220 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'142 2020-03-11T22:53:33.220 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'143 2020-03-11T22:53:33.220 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'144 2020-03-11T22:53:33.221 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'145 2020-03-11T22:53:33.221 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'146 2020-03-11T22:53:33.221 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'147 2020-03-11T22:53:33.221 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'148 2020-03-11T22:53:33.221 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'149 2020-03-11T22:53:33.221 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:28.320+0000 7f7819a51700 20 copy_up_to/copy_after copy dup from log version 30'150 2020-03-11T22:53:33.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:53:33.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:53:33.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:33.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:33.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:33.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:33.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:33.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:33.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:33.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:33.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:33.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:33.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:33.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:33.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:33.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:33.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:33.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:33.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:33.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:33.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:33.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:33.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:33.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:33.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:33.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:33.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:33.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76578 2020-03-11T22:53:33.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:33.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:33.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:53:33.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:53:33.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:33.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:33.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:33.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:33.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:33.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:33.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:33.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:33.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:33.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:33.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:33.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:33.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:33.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:33.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:33.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:33.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:33.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:33.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:33.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:33.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76578 2020-03-11T22:53:33.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:33.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:33.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:53:33.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:53:33.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:33.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:33.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76578 2020-03-11T22:53:33.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:53:33.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:53:33.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:53:33.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:53:33.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-03-11T22:53:33.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:111: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-03-11T22:53:33.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:33.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:53:33.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-03-11T22:53:33.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:53:33.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:53:33.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:53:33.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:53:33.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:53:33.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:53:33.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:53:33.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:53:33.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:53:33.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:33.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:33.558 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:33.558 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:33.558 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:33.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:33.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:53:33.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:53:33.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:53:33.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:33.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:33.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:53:33.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:53:33.720 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:33.720 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:33.720 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:33.720 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:33.720 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:33.720 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:33.720 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76578/ceph-mon.a.asok 2020-03-11T22:53:33.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:33.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76578/ceph-mon.a.asok config get fsid 2020-03-11T22:53:33.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:53:33.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:33.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:33.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:53:33.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:53:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76578/ceph-mon.a.asok 2020-03-11T22:53:33.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:33.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76578/ceph-mon.a.asok config get mon_host 2020-03-11T22:53:33.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:53:33.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:33.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:53:33.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:53:33.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:53:33.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:53:33.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:53:34.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:53:34.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:34.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:34.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:34.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:34.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:34.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:34.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:53:34.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:53:34.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:53:34.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:34.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:34.389 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:53:34.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:53:34.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:34.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:53:34.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:34.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:34.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:53:34.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:34.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:53:34.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:34.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:34.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:34.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:34.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:53:34.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:53:34.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:34.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:34.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:34.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:34.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:34.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:34.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:34.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:53:34.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:34.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2ddf6ab6-8ab4-408b-89c9-b4b735115937 2020-03-11T22:53:34.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 2ddf6ab6-8ab4-408b-89c9-b4b735115937' 2020-03-11T22:53:34.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:34.399 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 2ddf6ab6-8ab4-408b-89c9-b4b735115937 2020-03-11T22:53:34.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDua2leReSmGBAAdpMlsv2KOjROebPJY3A5HA== 2020-03-11T22:53:34.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDua2leReSmGBAAdpMlsv2KOjROebPJY3A5HA=="}' 2020-03-11T22:53:34.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2ddf6ab6-8ab4-408b-89c9-b4b735115937 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:53:34.741 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:53:34.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:53:34.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDua2leReSmGBAAdpMlsv2KOjROebPJY3A5HA== --osd-uuid 2ddf6ab6-8ab4-408b-89c9-b4b735115937 2020-03-11T22:53:34.775 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:34.771+0000 7fc62cfa6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:34.776 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:34.773+0000 7fc62cfa6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:34.779 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:34.775+0000 7fc62cfa6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:34.779 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:34.776+0000 7fc62cfa6ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:34.780 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:34.776+0000 7fc62cfa6ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:34.780 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:34.776+0000 7fc62cfa6ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:34.823 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:34.819+0000 7fc62cfa6ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:53:37.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:53:37.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:37.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:53:37.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:37.829 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:53:38.281 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:53:38.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:53:38.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:38.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:38.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:38.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:38.302 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:38.299+0000 7f6c11ec0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:38.306 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:38.302+0000 7f6c11ec0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:38.308 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:38.304+0000 7f6c11ec0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:38.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:53:38.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:38.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:53:38.823 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:38.819+0000 7f6c11ec0ec0 -1 Falling back to public interface 2020-03-11T22:53:38.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:39.554 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:39.550+0000 7f6c11ec0ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:53:39.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:39.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:39.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:39.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:39.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:53:39.994 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:53:40.344 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85131,v1:127.0.0.1:6803/85131] [v2:127.0.0.1:6804/85131,v1:127.0.0.1:6805/85131] exists,up 2ddf6ab6-8ab4-408b-89c9-b4b735115937 2020-03-11T22:53:40.344 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 07f3302c-9afd-4f12-8001-bdbbb8a3a5a1 2020-03-11T22:53:40.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:40.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:40.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:40.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:40.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:53:40.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:40.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:40.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:53:40.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:40.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:53:40.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:40.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:40.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:40.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:40.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:53:40.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:53:40.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:40.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:40.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:40.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:40.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:40.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:40.348 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:40.348 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:40.348 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:40.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:40.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:40.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:40.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:40.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:40.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:40.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:40.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:40.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:40.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:40.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:40.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:40.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:53:40.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:40.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=07f3302c-9afd-4f12-8001-bdbbb8a3a5a1 2020-03-11T22:53:40.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 07f3302c-9afd-4f12-8001-bdbbb8a3a5a1' 2020-03-11T22:53:40.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:40.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD0a2leTEAwFRAAELPUi4rmvJ4eDVnbplo7tA== 2020-03-11T22:53:40.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD0a2leTEAwFRAAELPUi4rmvJ4eDVnbplo7tA=="}' 2020-03-11T22:53:40.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 07f3302c-9afd-4f12-8001-bdbbb8a3a5a1 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:53:40.886 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:53:40.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:53:40.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD0a2leTEAwFRAAELPUi4rmvJ4eDVnbplo7tA== --osd-uuid 07f3302c-9afd-4f12-8001-bdbbb8a3a5a1 2020-03-11T22:53:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:40.916+0000 7f4e351f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:40.944 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:40.941+0000 7f4e351f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:40.946 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:40.942+0000 7f4e351f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:40.947 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:40.943+0000 7f4e351f9ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:40.947 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:40.943+0000 7f4e351f9ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:40.947 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:40.943+0000 7f4e351f9ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:41.011 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:41.008+0000 7f4e351f9ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:53:44.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:53:44.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:44.012 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:53:44.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:53:44.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:44.466 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:53:44.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:53:44.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:44.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:44.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:44.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:44.488 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:44.484+0000 7f01479ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:44.489 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:44.486+0000 7f01479ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:44.491 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:44.488+0000 7f01479ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:44.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:53:44.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:44.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:53:44.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:44.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:44.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:44.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:44.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:44.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:53:44.833 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:53:45.007 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:45.004+0000 7f01479ebec0 -1 Falling back to public interface 2020-03-11T22:53:45.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:45.825 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:45.821+0000 7f01479ebec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:53:46.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:46.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:46.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:46.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:46.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:53:46.179 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:53:46.561 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/85486,v1:127.0.0.1:6811/85486] [v2:127.0.0.1:6812/85486,v1:127.0.0.1:6813/85486] exists,up 07f3302c-9afd-4f12-8001-bdbbb8a3a5a1 2020-03-11T22:53:46.562 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 9ecb2767-3a50-4b94-9d26-10326831b038 2020-03-11T22:53:46.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:46.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:46.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:46.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:46.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:53:46.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:46.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:46.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:53:46.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:46.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:53:46.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:46.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:46.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:46.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:46.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:53:46.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:53:46.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:46.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:46.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:46.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:46.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:46.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:46.565 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:46.565 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:46.565 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:46.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:46.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:46.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:46.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:46.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:46.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:46.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:46.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:46.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:46.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:46.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:46.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:46.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:53:46.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:46.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9ecb2767-3a50-4b94-9d26-10326831b038 2020-03-11T22:53:46.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 9ecb2767-3a50-4b94-9d26-10326831b038' 2020-03-11T22:53:46.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:46.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD6a2leF9kXIhAA0HMj2uuxROt69gFKybhmiQ== 2020-03-11T22:53:46.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD6a2leF9kXIhAA0HMj2uuxROt69gFKybhmiQ=="}' 2020-03-11T22:53:46.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9ecb2767-3a50-4b94-9d26-10326831b038 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:53:47.037 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:53:47.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:53:47.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD6a2leF9kXIhAA0HMj2uuxROt69gFKybhmiQ== --osd-uuid 9ecb2767-3a50-4b94-9d26-10326831b038 2020-03-11T22:53:47.070 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:47.067+0000 7fd8a1be9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:47.072 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:47.068+0000 7fd8a1be9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:47.074 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:47.070+0000 7fd8a1be9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:47.075 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:47.071+0000 7fd8a1be9ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:47.075 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:47.071+0000 7fd8a1be9ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:47.075 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:47.071+0000 7fd8a1be9ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:47.165 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:47.162+0000 7fd8a1be9ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:53:50.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:53:50.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:50.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:53:50.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:50.342 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:53:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:53:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:50.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:50.800 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:53:50.820 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:50.817+0000 7efe462a4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:50.822 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:50.819+0000 7efe462a4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:50.823 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:50.820+0000 7efe462a4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:51.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:53:51.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:51.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:53:51.340 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:51.337+0000 7efe462a4ec0 -1 Falling back to public interface 2020-03-11T22:53:51.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:52.121 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:52.117+0000 7efe462a4ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:53:52.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:52.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:52.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:52.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:52.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:53:52.514 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:53:52.871 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/85841,v1:127.0.0.1:6819/85841] [v2:127.0.0.1:6820/85841,v1:127.0.0.1:6821/85841] exists,up 9ecb2767-3a50-4b94-9d26-10326831b038 2020-03-11T22:53:52.872 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 223c05bb-b443-4610-ace6-f1a783baa129 2020-03-11T22:53:52.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:52.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:52.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:52.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:52.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:53:52.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:52.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:52.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:53:52.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:52.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:53:52.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:52.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:52.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:52.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:52.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:53:52.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:53:52.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:52.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:52.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:52.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:52.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:52.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:52.877 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:52.877 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:52.878 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:52.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:52.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:52.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:52.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:52.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:52.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:52.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:52.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:53:52.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:52.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=223c05bb-b443-4610-ace6-f1a783baa129 2020-03-11T22:53:52.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 223c05bb-b443-4610-ace6-f1a783baa129' 2020-03-11T22:53:52.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:52.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAAbGleSHJrNBAA42vhdIQ2SPjburQUWGoolg== 2020-03-11T22:53:52.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAAbGleSHJrNBAA42vhdIQ2SPjburQUWGoolg=="}' 2020-03-11T22:53:52.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 223c05bb-b443-4610-ace6-f1a783baa129 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:53:53.404 INFO:tasks.workunit.client.0.smithi130.stdout:3 2020-03-11T22:53:53.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:53:53.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAAbGleSHJrNBAA42vhdIQ2SPjburQUWGoolg== --osd-uuid 223c05bb-b443-4610-ace6-f1a783baa129 2020-03-11T22:53:53.443 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:53.435+0000 7fdc21bf2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:53.443 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:53.437+0000 7fdc21bf2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:53.443 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:53.439+0000 7fdc21bf2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:53.444 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:53.439+0000 7fdc21bf2ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:53.444 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:53.439+0000 7fdc21bf2ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:53.444 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:53.439+0000 7fdc21bf2ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:53.503 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:53.500+0000 7fdc21bf2ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:53:56.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:53:56.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:56.629 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2020-03-11T22:53:56.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:53:56.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:57.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:53:57.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:57.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:57.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:57.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:57.085 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2020-03-11T22:53:57.104 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:57.102+0000 7f12aca32ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:57.106 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:57.103+0000 7f12aca32ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:57.108 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:57.106+0000 7f12aca32ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:57.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:53:57.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:57.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:53:57.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:57.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:57.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:57.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:57.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:57.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:57.431 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:53:57.625 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:57.622+0000 7f12aca32ec0 -1 Falling back to public interface 2020-03-11T22:53:57.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:58.394 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:58.391+0000 7f12aca32ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:53:58.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:58.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:58.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:58.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:58.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:58.825 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:53:59.205 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/86196,v1:127.0.0.1:6827/86196] [v2:127.0.0.1:6828/86196,v1:127.0.0.1:6829/86196] exists,up 223c05bb-b443-4610-ace6-f1a783baa129 2020-03-11T22:53:59.205 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 73ec9fbd-51a9-4bec-b579-ee7080a3eb21 2020-03-11T22:53:59.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:53:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:53:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:53:59.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:53:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:53:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:59.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:59.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:59.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:53:59.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:53:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:59.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:53:59.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=73ec9fbd-51a9-4bec-b579-ee7080a3eb21 2020-03-11T22:53:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 73ec9fbd-51a9-4bec-b579-ee7080a3eb21' 2020-03-11T22:53:59.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:59.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAHbGleDoDJDBAAlwLo5MAPhBKWEMpx0E320A== 2020-03-11T22:53:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAHbGleDoDJDBAAlwLo5MAPhBKWEMpx0E320A=="}' 2020-03-11T22:53:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 73ec9fbd-51a9-4bec-b579-ee7080a3eb21 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:53:59.758 INFO:tasks.workunit.client.0.smithi130.stdout:4 2020-03-11T22:53:59.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:53:59.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAHbGleDoDJDBAAlwLo5MAPhBKWEMpx0E320A== --osd-uuid 73ec9fbd-51a9-4bec-b579-ee7080a3eb21 2020-03-11T22:53:59.792 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:59.789+0000 7f2883b3dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:59.793 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:59.791+0000 7f2883b3dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:59.795 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:59.792+0000 7f2883b3dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:59.796 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:59.793+0000 7f2883b3dec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:59.796 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:59.793+0000 7f2883b3dec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:59.796 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:59.793+0000 7f2883b3dec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:59.867 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:53:59.864+0000 7f2883b3dec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:54:02.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:54:02.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:02.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:54:02.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:02.979 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2020-03-11T22:54:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:54:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:03.439 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2020-03-11T22:54:03.459 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:03.456+0000 7f6662932ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:03.461 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:03.458+0000 7f6662932ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:03.462 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:03.460+0000 7f6662932ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:03.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:54:03.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:03.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:54:03.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:03.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:03.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:03.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:03.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:03.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:03.831 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:54:03.980 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:03.978+0000 7f6662932ec0 -1 Falling back to public interface 2020-03-11T22:54:04.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:04.749 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:04.747+0000 7f6662932ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:54:05.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:05.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:05.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:05.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:05.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:05.188 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/86551,v1:127.0.0.1:6835/86551] [v2:127.0.0.1:6836/86551,v1:127.0.0.1:6837/86551] exists,up 73ec9fbd-51a9-4bec-b579-ee7080a3eb21 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 41824027-9c87-4e07-a3a9-7c0dbe92723b 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=41824027-9c87-4e07-a3a9-7c0dbe92723b 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 41824027-9c87-4e07-a3a9-7c0dbe92723b' 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQANbGleHILHIhAAzzif3zEXmU3y3IQXads01Q== 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQANbGleHILHIhAAzzif3zEXmU3y3IQXads01Q=="}' 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 41824027-9c87-4e07-a3a9-7c0dbe92723b -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:06.062 INFO:tasks.workunit.client.0.smithi130.stdout:5 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQANbGleHILHIhAAzzif3zEXmU3y3IQXads01Q== --osd-uuid 41824027-9c87-4e07-a3a9-7c0dbe92723b 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:06.093+0000 7f3604399ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:06.097 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:06.095+0000 7f3604399ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:06.097+0000 7f3604399ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:06.097+0000 7f3604399ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:06.097+0000 7f3604399ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:06.097+0000 7f3604399ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:06.198 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:06.195+0000 7f3604399ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:54:09.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:54:09.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:09.324 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2020-03-11T22:54:09.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:54:09.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:09.783 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2020-03-11T22:54:09.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:54:09.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:09.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:09.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:09.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:09.805 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:09.801+0000 7fcd8fcf0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:09.806 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:09.802+0000 7fcd8fcf0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:09.808 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:09.804+0000 7fcd8fcf0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:10.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:54:10.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:10.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:54:10.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:10.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:10.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:10.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:10.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:10.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:10.144 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:54:10.323 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:10.319+0000 7fcd8fcf0ec0 -1 Falling back to public interface 2020-03-11T22:54:10.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:11.081 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:54:11.077+0000 7fcd8fcf0ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:54:11.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:11.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:11.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:11.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:11.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:11.505 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:54:11.932 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/86914,v1:127.0.0.1:6843/86914] [v2:127.0.0.1:6844/86914,v1:127.0.0.1:6845/86914] exists,up 41824027-9c87-4e07-a3a9-7c0dbe92723b 2020-03-11T22:54:11.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:11.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:11.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:11.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:54:11.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:54:12.433 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2020-03-11T22:54:12.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:54:13.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:54:13.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:13.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:54:13.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:13.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:54:13.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:13.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:54:13.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:13.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:54:13.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:13.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:54:13.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:13.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:54:14.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:54:14.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:54:14.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:54:14.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:54:14.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:54:14.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:54:14.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:54:14.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:54:14.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:54:14.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:54:14.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:54:14.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:54:14.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:54:14.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:14.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:54:15.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:54:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:54:15.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:54:15.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:54:15.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:54:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:54:15.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:54:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:54:15.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:54:15.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:54:15.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:54:15.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:54:15.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:54:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:54:16.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:54:16.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:54:16.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:54:16.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:54:16.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:54:16.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:54:16.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:54:16.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:54:16.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:54:16.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:54:16.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:54:16.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:54:16.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:54:16.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:54:16.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:16.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:54:17.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:54:17.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:54:17.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:54:17.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:54:17.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:54:17.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:54:17.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:54:17.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:54:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:54:17.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:54:17.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:54:17.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:54:17.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:54:17.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:17.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:54:18.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:54:18.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:54:18.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:54:18.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:54:18.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:54:18.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:54:18.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:54:18.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:54:18.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:54:18.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:54:18.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:54:18.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:54:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:54:18.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:54:18.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:18.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:54:19.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:54:19.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:54:19.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:54:19.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:54:19.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:54:19.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:54:19.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:54:19.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:54:19.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:54:19.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:54:19.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:54:19.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:54:19.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:54:19.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:54:19.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:19.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:54:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:54:20.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:54:20.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:54:20.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:54:20.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:54:20.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:54:20.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:54:20.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:54:20.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:54:20.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:54:20.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:54:20.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:54:20.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:54:20.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:20.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:54:21.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:54:21.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:54:21.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:54:21.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:54:21.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:54:21.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:54:21.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:54:21.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:54:21.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:54:21.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:54:21.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:54:21.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:54:21.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:54:21.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:21.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:54:22.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:54:22.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:54:22.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:54:22.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:54:22.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:54:22.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:54:22.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:54:22.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:54:22.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:54:22.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:54:22.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:54:22.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:54:22.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:54:22.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:54:22.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:22.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:54:23.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:54:23.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:54:23.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:54:23.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:54:23.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:54:23.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:54:23.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:54:23.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:54:23.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:54:23.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:54:23.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:54:23.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:54:23.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:23.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:54:24.049 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:54:24.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:54:24.549 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:54:24.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:24.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:54:25.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:54:25.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:54:25.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:25.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:25.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:25.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:25.561 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:25.561 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:25.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:25.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:25.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:25.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:25.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:25.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:25.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:25.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:26.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:26.006 INFO:tasks.workunit.client.0.smithi130.stderr:1 2020-03-11T22:54:26.006 INFO:tasks.workunit.client.0.smithi130.stderr:2 2020-03-11T22:54:26.006 INFO:tasks.workunit.client.0.smithi130.stderr:3 2020-03-11T22:54:26.006 INFO:tasks.workunit.client.0.smithi130.stderr:4 2020-03-11T22:54:26.007 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2020-03-11T22:54:26.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:26.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:26.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:26.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:54:26.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:54:26.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:26.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:26.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672970 2020-03-11T22:54:26.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2020-03-11T22:54:26.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:26.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:26.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542154 2020-03-11T22:54:26.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542154' 2020-03-11T22:54:26.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:26.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:26.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378632 2020-03-11T22:54:26.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542154 3-81604378632' 2020-03-11T22:54:26.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:26.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:26.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215112 2020-03-11T22:54:26.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542154 3-81604378632 4-103079215112' 2020-03-11T22:54:26.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:26.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:54:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=120259084293 2020-03-11T22:54:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542154 3-81604378632 4-103079215112 5-120259084293' 2020-03-11T22:54:26.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:26.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:54:26.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:26.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:26.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:54:26.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:26.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:54:26.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:54:26.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:26.749 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:54:27.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836492 2020-03-11T22:54:27.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:28.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:28.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:28.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:54:28.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:28.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672970 2020-03-11T22:54:28.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:28.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:28.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672970 2020-03-11T22:54:28.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:28.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672970 2020-03-11T22:54:28.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-03-11T22:54:28.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:28.451 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672970 2020-03-11T22:54:28.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672970 2020-03-11T22:54:28.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:28.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542154 2020-03-11T22:54:28.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:28.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:28.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542154 2020-03-11T22:54:28.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:28.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542154 2020-03-11T22:54:28.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2020-03-11T22:54:28.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:28.802 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 60129542154 2020-03-11T22:54:29.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542154 -lt 60129542154 2020-03-11T22:54:29.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:29.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378632 2020-03-11T22:54:29.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:29.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:29.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378632 2020-03-11T22:54:29.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:29.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378632 2020-03-11T22:54:29.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378632' 2020-03-11T22:54:29.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:29.183 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 81604378632 2020-03-11T22:54:29.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378632 -lt 81604378632 2020-03-11T22:54:29.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:29.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215112 2020-03-11T22:54:29.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:29.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:29.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215112 2020-03-11T22:54:29.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:29.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215112 2020-03-11T22:54:29.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215112' 2020-03-11T22:54:29.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:29.534 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 103079215112 2020-03-11T22:54:29.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215112 -lt 103079215112 2020-03-11T22:54:29.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:29.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-120259084293 2020-03-11T22:54:29.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:54:29.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-120259084293 2020-03-11T22:54:29.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:29.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=120259084293 2020-03-11T22:54:29.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 120259084293' 2020-03-11T22:54:29.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:54:29.881 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 120259084293 2020-03-11T22:54:30.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 120259084294 -lt 120259084293 2020-03-11T22:54:30.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:54:30.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:30.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:30.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:54:30.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:30.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:30.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:30.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:30.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:30.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:30.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:31.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:31.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:31.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:31.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:31.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:31.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:54:31.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:54:31.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:54:31.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:31.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:31.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:31.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:31.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:31.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:31.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:31.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:31.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:31.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:31.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:31.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:31.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:31.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:32.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:32.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:32.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:32.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:32.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:32.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:32.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:32.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:32.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:32.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26200 2020-03-11T22:54:32.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26200 '!=' null 2020-03-11T22:54:32.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:32.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:32.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:32.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:32.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:32.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:32.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:32.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:32.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:32.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:32.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:33.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:33.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:33.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:33.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:33.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:33.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:33.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:33.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:33.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:33.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:33.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:33.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:33.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:34.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20971 2020-03-11T22:54:34.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20971 '!=' null 2020-03-11T22:54:34.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:34.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:34.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:34.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:34.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:34.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:34.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:34.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:34.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:34.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:34.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:34.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:34.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:34.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:34.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:35.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:35.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:35.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:35.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:35.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:35.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:35.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:35.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:35.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:35.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20971 2020-03-11T22:54:35.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20971 '!=' null 2020-03-11T22:54:35.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:35.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:35.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:35.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:35.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:35.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:35.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:35.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:35.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:35.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:35.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:36.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:36.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:36.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:36.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:36.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:36.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:36.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:36.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17770 2020-03-11T22:54:36.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17770 '!=' null 2020-03-11T22:54:36.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:36.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:36.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:37.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:37.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:37.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:37.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:37.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:37.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:37.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:37.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:37.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:37.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:37.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:37.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:37.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:37.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:37.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:37.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:37.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:37.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:37.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:37.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:37.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:38.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:54:38.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:54:38.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:38.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:38.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:38.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:38.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:38.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:38.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:38.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:38.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:38.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:38.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:38.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:38.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:38.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:38.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:39.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:39.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:39.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:39.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:39.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:39.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:39.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:39.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:39.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:39.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:54:39.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:54:39.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:39.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:39.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:39.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:39.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:39.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:39.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:39.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:39.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:39.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:39.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:40.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:40.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:40.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:40.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:40.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:40.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:40.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30474 2020-03-11T22:54:40.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30474 '!=' null 2020-03-11T22:54:40.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:41.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:41.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:41.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:41.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:41.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:41.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:41.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:41.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:41.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:41.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:41.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:41.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:41.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:41.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:41.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:41.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:41.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:41.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:41.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:41.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:41.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:42.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=46573 2020-03-11T22:54:42.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 46573 '!=' null 2020-03-11T22:54:42.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:42.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:42.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:42.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:42.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:42.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:42.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:42.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:42.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:42.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:42.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:42.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:42.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:42.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:42.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:43.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:43.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:43.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:43.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:43.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:43.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:43.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:43.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:43.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:43.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=46573 2020-03-11T22:54:43.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 46573 '!=' null 2020-03-11T22:54:43.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:43.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:43.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:43.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:43.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:43.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:43.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:43.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:43.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:43.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:43.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:44.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:44.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:44.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:44.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:44.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:44.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:44.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:44.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:44.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:44.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:44.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:44.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:44.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:45.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:54:45.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:54:45.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:45.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:45.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:45.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:45.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:45.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:45.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:45.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:45.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:45.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:45.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:45.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:45.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:45.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:45.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:45.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:45.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:45.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:45.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:45.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:45.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:45.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:45.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:45.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:46.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:54:46.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:54:46.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:46.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:46.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:46.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:46.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:46.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:46.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:46.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:46.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:46.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:46.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:46.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:46.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:46.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:46.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:47.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:47.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:47.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:47.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:47.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:47.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:47.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:47.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:47.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:47.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:54:47.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:54:47.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:47.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:47.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:47.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:47.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:47.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:47.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:47.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:47.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:47.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:47.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:48.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:48.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:48.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:48.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:48.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:48.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:48.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:48.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:48.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:48.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:48.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:48.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:48.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:49.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=48298 2020-03-11T22:54:49.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 48298 '!=' null 2020-03-11T22:54:49.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:49.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:49.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:49.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:49.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:49.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:49.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:49.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:49.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:49.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:49.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:49.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:49.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:49.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:49.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:49.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:49.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:49.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:49.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:49.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:49.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:50.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32773 2020-03-11T22:54:50.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32773 '!=' null 2020-03-11T22:54:50.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:50.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:50.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:50.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:50.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:50.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:50.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:50.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:50.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:50.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:50.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:50.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:50.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:50.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:50.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:51.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:51.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:51.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:51.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:51.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:51.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:51.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:51.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:51.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:51.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=49448 2020-03-11T22:54:51.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 49448 '!=' null 2020-03-11T22:54:51.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:51.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:51.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:51.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:51.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:51.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:51.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:51.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:51.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:51.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:51.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:52.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:52.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:52.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:52.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:52.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:52.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:52.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:52.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:52.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:52.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:52.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:52.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:52.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:53.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=49448 2020-03-11T22:54:53.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 49448 '!=' null 2020-03-11T22:54:53.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:53.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:53.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:53.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:53.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:53.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:53.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:53.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:53.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:53.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:53.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:53.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:53.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:53.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:53.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:54.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:54.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:54.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:54.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:54.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=50022 2020-03-11T22:54:54.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 50022 '!=' null 2020-03-11T22:54:54.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:54.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:54.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:54.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:54.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:54.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:54.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:54.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:54.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:54.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:54.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:55.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:55.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:55.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:55.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:55.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:55.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:55.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:55.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:55.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:55.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:55.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:55.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:55.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:55.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25010 2020-03-11T22:54:55.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25010 '!=' null 2020-03-11T22:54:55.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:55.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:55.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:56.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:56.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:56.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:56.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:56.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:56.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:56.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:56.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:56.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:56.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:56.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:56.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:56.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:56.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:56.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:56.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:56.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:56.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:56.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:56.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:56.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:57.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25010 2020-03-11T22:54:57.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25010 '!=' null 2020-03-11T22:54:57.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:57.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:57.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:57.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:57.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:57.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:57.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:57.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:57.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:58.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:58.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:58.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:58.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:58.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:58.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:58.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:58.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:58.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:58.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34497 2020-03-11T22:54:58.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34497 '!=' null 2020-03-11T22:54:58.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:58.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:58.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:58.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:58.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:58.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:58.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:58.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:58.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:58.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:58.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:58.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:58.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:58.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:58.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:59.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:54:59.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:54:59.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:54:59.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:59.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:54:59.773 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2020-03-11T22:54:59.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:54:59.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:54:59.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:59.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:59.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:59.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:59.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:00.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:00.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:55:00.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:55:00.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:55:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:55:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:55:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:55:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:55:00.727 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2020-03-11T22:55:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_log_t": { 2020-03-11T22:55:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "head": "31'150", 2020-03-11T22:55:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "tail": "31'148", 2020-03-11T22:55:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "log": [ 2020-03-11T22:55:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2020-03-11T22:55:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:55:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'149", 2020-03-11T22:55:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2020-03-11T22:55:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4631.0:1", 2020-03-11T22:55:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2020-03-11T22:55:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2020-03-11T22:54:23.663428+0000", 2020-03-11T22:55:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2020-03-11T22:55:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2020-03-11T22:55:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2020-03-11T22:55:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2020-03-11T22:55:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2020-03-11T22:55:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2020-03-11T22:55:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2020-03-11T22:55:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2020-03-11T22:55:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2020-03-11T22:55:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2020-03-11T22:55:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'150", 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2020-03-11T22:55:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4634.0:1", 2020-03-11T22:55:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2020-03-11T22:55:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2020-03-11T22:54:23.704840+0000", 2020-03-11T22:55:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2020-03-11T22:55:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2020-03-11T22:55:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2020-03-11T22:55:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2020-03-11T22:55:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2020-03-11T22:55:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2020-03-11T22:55:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2020-03-11T22:55:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2020-03-11T22:55:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2020-03-11T22:55:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2020-03-11T22:55:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2020-03-11T22:55:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "dups": [ 2020-03-11T22:55:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4187.0:1", 2020-03-11T22:55:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "30'1", 2020-03-11T22:55:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "1", 2020-03-11T22:55:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4190.0:1", 2020-03-11T22:55:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'2", 2020-03-11T22:55:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "2", 2020-03-11T22:55:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4193.0:1", 2020-03-11T22:55:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'3", 2020-03-11T22:55:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "3", 2020-03-11T22:55:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4196.0:1", 2020-03-11T22:55:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'4", 2020-03-11T22:55:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "4", 2020-03-11T22:55:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4199.0:1", 2020-03-11T22:55:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'5", 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "5", 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4202.0:1", 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'6", 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "6", 2020-03-11T22:55:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.741 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.741 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.741 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4205.0:1", 2020-03-11T22:55:00.741 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'7", 2020-03-11T22:55:00.741 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "7", 2020-03-11T22:55:00.741 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.747 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.747 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.747 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4208.0:1", 2020-03-11T22:55:00.747 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'8", 2020-03-11T22:55:00.748 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "8", 2020-03-11T22:55:00.748 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.748 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.748 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.748 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4211.0:1", 2020-03-11T22:55:00.748 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'9", 2020-03-11T22:55:00.749 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "9", 2020-03-11T22:55:00.749 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.749 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.749 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.749 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4214.0:1", 2020-03-11T22:55:00.749 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'10", 2020-03-11T22:55:00.749 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "10", 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4217.0:1", 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'11", 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "11", 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.750 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.751 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.751 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4220.0:1", 2020-03-11T22:55:00.751 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'12", 2020-03-11T22:55:00.751 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "12", 2020-03-11T22:55:00.751 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.751 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.751 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.752 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4223.0:1", 2020-03-11T22:55:00.752 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'13", 2020-03-11T22:55:00.752 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "13", 2020-03-11T22:55:00.752 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.752 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.752 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.752 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4226.0:1", 2020-03-11T22:55:00.753 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'14", 2020-03-11T22:55:00.753 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "14", 2020-03-11T22:55:00.753 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.753 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.753 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.753 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4229.0:1", 2020-03-11T22:55:00.753 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'15", 2020-03-11T22:55:00.754 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "15", 2020-03-11T22:55:00.754 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.754 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.754 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.754 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4232.0:1", 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'16", 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "16", 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4235.0:1", 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'17", 2020-03-11T22:55:00.755 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "17", 2020-03-11T22:55:00.756 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.756 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.756 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.756 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4238.0:1", 2020-03-11T22:55:00.756 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'18", 2020-03-11T22:55:00.756 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "18", 2020-03-11T22:55:00.756 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.757 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.757 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.758 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4241.0:1", 2020-03-11T22:55:00.758 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'19", 2020-03-11T22:55:00.758 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "19", 2020-03-11T22:55:00.758 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.758 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.758 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.759 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4244.0:1", 2020-03-11T22:55:00.759 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'20", 2020-03-11T22:55:00.759 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "20", 2020-03-11T22:55:00.759 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.759 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.759 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.759 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4247.0:1", 2020-03-11T22:55:00.760 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'21", 2020-03-11T22:55:00.760 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "21", 2020-03-11T22:55:00.760 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.760 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.760 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.760 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4250.0:1", 2020-03-11T22:55:00.760 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'22", 2020-03-11T22:55:00.761 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "22", 2020-03-11T22:55:00.761 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.761 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.761 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.761 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4253.0:1", 2020-03-11T22:55:00.761 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'23", 2020-03-11T22:55:00.763 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "23", 2020-03-11T22:55:00.763 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.763 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.763 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.763 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4256.0:1", 2020-03-11T22:55:00.764 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'24", 2020-03-11T22:55:00.764 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "24", 2020-03-11T22:55:00.764 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.764 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.764 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.765 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4259.0:1", 2020-03-11T22:55:00.765 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'25", 2020-03-11T22:55:00.765 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "25", 2020-03-11T22:55:00.765 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.765 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.766 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4262.0:1", 2020-03-11T22:55:00.766 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'26", 2020-03-11T22:55:00.766 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "26", 2020-03-11T22:55:00.766 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.766 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.766 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.767 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4265.0:1", 2020-03-11T22:55:00.767 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'27", 2020-03-11T22:55:00.767 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "27", 2020-03-11T22:55:00.767 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.767 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.767 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.767 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4268.0:1", 2020-03-11T22:55:00.768 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'28", 2020-03-11T22:55:00.769 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "28", 2020-03-11T22:55:00.769 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.769 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.769 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4271.0:1", 2020-03-11T22:55:00.769 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'29", 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "29", 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4274.0:1", 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'30", 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "30", 2020-03-11T22:55:00.770 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.771 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.771 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.771 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4277.0:1", 2020-03-11T22:55:00.771 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'31", 2020-03-11T22:55:00.771 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "31", 2020-03-11T22:55:00.771 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.771 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4280.0:1", 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'32", 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "32", 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.772 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4283.0:1", 2020-03-11T22:55:00.773 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'33", 2020-03-11T22:55:00.773 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "33", 2020-03-11T22:55:00.773 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.773 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.773 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.773 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4286.0:1", 2020-03-11T22:55:00.773 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'34", 2020-03-11T22:55:00.774 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "34", 2020-03-11T22:55:00.774 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.774 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.774 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.774 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4289.0:1", 2020-03-11T22:55:00.774 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'35", 2020-03-11T22:55:00.774 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "35", 2020-03-11T22:55:00.775 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.775 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.775 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.775 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4292.0:1", 2020-03-11T22:55:00.775 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'36", 2020-03-11T22:55:00.775 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "36", 2020-03-11T22:55:00.775 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4295.0:1", 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'37", 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "37", 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.776 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.777 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4298.0:1", 2020-03-11T22:55:00.777 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'38", 2020-03-11T22:55:00.777 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "38", 2020-03-11T22:55:00.777 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.777 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.777 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.778 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4301.0:1", 2020-03-11T22:55:00.778 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'39", 2020-03-11T22:55:00.778 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "39", 2020-03-11T22:55:00.778 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.778 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.778 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.778 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4304.0:1", 2020-03-11T22:55:00.779 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'40", 2020-03-11T22:55:00.779 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "40", 2020-03-11T22:55:00.779 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.779 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.779 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.779 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4307.0:1", 2020-03-11T22:55:00.779 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'41", 2020-03-11T22:55:00.780 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "41", 2020-03-11T22:55:00.780 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.780 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.780 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.780 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4310.0:1", 2020-03-11T22:55:00.780 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'42", 2020-03-11T22:55:00.781 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "42", 2020-03-11T22:55:00.781 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.781 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.781 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.781 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4313.0:1", 2020-03-11T22:55:00.781 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'43", 2020-03-11T22:55:00.781 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "43", 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4316.0:1", 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'44", 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "44", 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.782 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.783 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.783 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4319.0:1", 2020-03-11T22:55:00.783 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'45", 2020-03-11T22:55:00.783 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "45", 2020-03-11T22:55:00.783 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.783 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.783 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.784 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4322.0:1", 2020-03-11T22:55:00.784 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'46", 2020-03-11T22:55:00.784 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "46", 2020-03-11T22:55:00.784 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.784 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.785 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.785 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4325.0:1", 2020-03-11T22:55:00.785 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'47", 2020-03-11T22:55:00.785 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "47", 2020-03-11T22:55:00.785 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.786 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.786 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.786 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4328.0:1", 2020-03-11T22:55:00.786 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'48", 2020-03-11T22:55:00.787 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "48", 2020-03-11T22:55:00.787 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.787 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.787 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.787 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4331.0:1", 2020-03-11T22:55:00.787 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'49", 2020-03-11T22:55:00.787 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "49", 2020-03-11T22:55:00.788 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.788 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.788 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.788 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4334.0:1", 2020-03-11T22:55:00.788 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'50", 2020-03-11T22:55:00.788 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "50", 2020-03-11T22:55:00.788 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.789 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.789 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.789 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4337.0:1", 2020-03-11T22:55:00.789 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'51", 2020-03-11T22:55:00.789 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "51", 2020-03-11T22:55:00.789 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.790 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.790 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.790 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4340.0:1", 2020-03-11T22:55:00.790 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'52", 2020-03-11T22:55:00.790 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "52", 2020-03-11T22:55:00.790 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.790 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4343.0:1", 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'53", 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "53", 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.791 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4346.0:1", 2020-03-11T22:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'54", 2020-03-11T22:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "54", 2020-03-11T22:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4349.0:1", 2020-03-11T22:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'55", 2020-03-11T22:55:00.793 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "55", 2020-03-11T22:55:00.793 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.793 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.793 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.793 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4352.0:1", 2020-03-11T22:55:00.793 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'56", 2020-03-11T22:55:00.793 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "56", 2020-03-11T22:55:00.794 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.794 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.794 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.794 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4355.0:1", 2020-03-11T22:55:00.794 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'57", 2020-03-11T22:55:00.794 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "57", 2020-03-11T22:55:00.794 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4358.0:1", 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'58", 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "58", 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.795 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.796 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4361.0:1", 2020-03-11T22:55:00.796 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'59", 2020-03-11T22:55:00.796 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "59", 2020-03-11T22:55:00.796 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.796 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.796 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.796 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4364.0:1", 2020-03-11T22:55:00.797 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'60", 2020-03-11T22:55:00.797 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "60", 2020-03-11T22:55:00.797 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.797 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.797 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.797 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4367.0:1", 2020-03-11T22:55:00.797 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'61", 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "61", 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4370.0:1", 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'62", 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "62", 2020-03-11T22:55:00.798 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.799 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.799 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.799 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4373.0:1", 2020-03-11T22:55:00.799 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'63", 2020-03-11T22:55:00.799 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "63", 2020-03-11T22:55:00.799 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.800 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.800 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.800 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4376.0:1", 2020-03-11T22:55:00.800 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'64", 2020-03-11T22:55:00.800 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "64", 2020-03-11T22:55:00.801 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.801 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.801 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.801 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4379.0:1", 2020-03-11T22:55:00.801 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'65", 2020-03-11T22:55:00.801 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "65", 2020-03-11T22:55:00.801 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4382.0:1", 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'66", 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "66", 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.802 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.803 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4385.0:1", 2020-03-11T22:55:00.803 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'67", 2020-03-11T22:55:00.803 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "67", 2020-03-11T22:55:00.803 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.803 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.803 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.803 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4388.0:1", 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'68", 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "68", 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4391.0:1", 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'69", 2020-03-11T22:55:00.804 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "69", 2020-03-11T22:55:00.805 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.805 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.805 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.805 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4394.0:1", 2020-03-11T22:55:00.805 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'70", 2020-03-11T22:55:00.805 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "70", 2020-03-11T22:55:00.805 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.806 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.806 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.806 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4397.0:1", 2020-03-11T22:55:00.806 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'71", 2020-03-11T22:55:00.806 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "71", 2020-03-11T22:55:00.806 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.806 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4400.0:1", 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'72", 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "72", 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.807 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4403.0:1", 2020-03-11T22:55:00.808 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'73", 2020-03-11T22:55:00.808 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "73", 2020-03-11T22:55:00.808 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.808 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.808 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.808 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4406.0:1", 2020-03-11T22:55:00.808 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'74", 2020-03-11T22:55:00.809 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "74", 2020-03-11T22:55:00.809 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.809 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.809 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.809 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4409.0:1", 2020-03-11T22:55:00.809 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'75", 2020-03-11T22:55:00.809 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "75", 2020-03-11T22:55:00.810 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.810 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.810 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.810 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4412.0:1", 2020-03-11T22:55:00.811 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'76", 2020-03-11T22:55:00.811 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "76", 2020-03-11T22:55:00.811 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.811 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.811 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.811 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4415.0:1", 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'77", 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "77", 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4418.0:1", 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'78", 2020-03-11T22:55:00.812 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "78", 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4421.0:1", 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'79", 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "79", 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.813 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.814 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.814 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4424.0:1", 2020-03-11T22:55:00.814 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'80", 2020-03-11T22:55:00.814 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "80", 2020-03-11T22:55:00.814 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.814 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.814 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.815 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4427.0:1", 2020-03-11T22:55:00.815 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'81", 2020-03-11T22:55:00.815 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "81", 2020-03-11T22:55:00.815 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.815 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.815 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.815 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4430.0:1", 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'82", 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "82", 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4433.0:1", 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'83", 2020-03-11T22:55:00.816 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "83", 2020-03-11T22:55:00.817 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.817 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.817 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.817 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4436.0:1", 2020-03-11T22:55:00.817 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'84", 2020-03-11T22:55:00.817 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "84", 2020-03-11T22:55:00.817 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.818 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.818 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.818 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4439.0:1", 2020-03-11T22:55:00.818 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'85", 2020-03-11T22:55:00.818 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "85", 2020-03-11T22:55:00.818 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.818 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4442.0:1", 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'86", 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "86", 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.819 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4445.0:1", 2020-03-11T22:55:00.820 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'87", 2020-03-11T22:55:00.820 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "87", 2020-03-11T22:55:00.820 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.820 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.820 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.820 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4448.0:1", 2020-03-11T22:55:00.820 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'88", 2020-03-11T22:55:00.821 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "88", 2020-03-11T22:55:00.821 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.821 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.821 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.821 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4451.0:1", 2020-03-11T22:55:00.821 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'89", 2020-03-11T22:55:00.822 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "89", 2020-03-11T22:55:00.822 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.822 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.822 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.822 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4454.0:1", 2020-03-11T22:55:00.822 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'90", 2020-03-11T22:55:00.822 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "90", 2020-03-11T22:55:00.823 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.823 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.823 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.823 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4457.0:1", 2020-03-11T22:55:00.823 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'91", 2020-03-11T22:55:00.823 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "91", 2020-03-11T22:55:00.826 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.826 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.826 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.826 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4460.0:1", 2020-03-11T22:55:00.826 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'92", 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "92", 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4463.0:1", 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'93", 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "93", 2020-03-11T22:55:00.827 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.828 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.828 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.828 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4466.0:1", 2020-03-11T22:55:00.828 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'94", 2020-03-11T22:55:00.828 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "94", 2020-03-11T22:55:00.828 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.828 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.829 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.829 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4469.0:1", 2020-03-11T22:55:00.829 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'95", 2020-03-11T22:55:00.829 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "95", 2020-03-11T22:55:00.829 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.829 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.829 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4472.0:1", 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'96", 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "96", 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4475.0:1", 2020-03-11T22:55:00.830 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'97", 2020-03-11T22:55:00.831 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "97", 2020-03-11T22:55:00.831 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.831 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.831 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.831 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4478.0:1", 2020-03-11T22:55:00.831 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'98", 2020-03-11T22:55:00.831 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "98", 2020-03-11T22:55:00.832 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.832 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.832 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.832 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4481.0:1", 2020-03-11T22:55:00.832 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'99", 2020-03-11T22:55:00.832 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "99", 2020-03-11T22:55:00.832 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.833 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.833 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.833 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4484.0:1", 2020-03-11T22:55:00.833 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'100", 2020-03-11T22:55:00.833 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "100", 2020-03-11T22:55:00.833 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.833 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4487.0:1", 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'101", 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "101", 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.834 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4490.0:1", 2020-03-11T22:55:00.835 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'102", 2020-03-11T22:55:00.835 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "102", 2020-03-11T22:55:00.835 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.835 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.835 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.835 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4493.0:1", 2020-03-11T22:55:00.835 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'103", 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "103", 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4496.0:1", 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'104", 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "104", 2020-03-11T22:55:00.836 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.837 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.837 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.837 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4499.0:1", 2020-03-11T22:55:00.837 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'105", 2020-03-11T22:55:00.837 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "105", 2020-03-11T22:55:00.837 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.837 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4502.0:1", 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'106", 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "106", 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.838 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4505.0:1", 2020-03-11T22:55:00.839 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'107", 2020-03-11T22:55:00.839 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "107", 2020-03-11T22:55:00.839 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.839 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.839 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.839 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4508.0:1", 2020-03-11T22:55:00.839 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'108", 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "108", 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4511.0:1", 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'109", 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "109", 2020-03-11T22:55:00.840 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.841 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.841 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.841 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4514.0:1", 2020-03-11T22:55:00.841 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'110", 2020-03-11T22:55:00.841 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "110", 2020-03-11T22:55:00.841 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.841 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4517.0:1", 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'111", 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "111", 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.842 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4520.0:1", 2020-03-11T22:55:00.843 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'112", 2020-03-11T22:55:00.843 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "112", 2020-03-11T22:55:00.843 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.843 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.843 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.843 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4523.0:1", 2020-03-11T22:55:00.843 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'113", 2020-03-11T22:55:00.844 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "113", 2020-03-11T22:55:00.844 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.844 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.844 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.844 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4526.0:1", 2020-03-11T22:55:00.844 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'114", 2020-03-11T22:55:00.844 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "114", 2020-03-11T22:55:00.845 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.845 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.845 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.845 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4529.0:1", 2020-03-11T22:55:00.845 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'115", 2020-03-11T22:55:00.845 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "115", 2020-03-11T22:55:00.845 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.846 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.846 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.846 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4532.0:1", 2020-03-11T22:55:00.846 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'116", 2020-03-11T22:55:00.846 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "116", 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4535.0:1", 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'117", 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "117", 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.847 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.848 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.848 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4538.0:1", 2020-03-11T22:55:00.848 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'118", 2020-03-11T22:55:00.848 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "118", 2020-03-11T22:55:00.848 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.848 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.848 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.849 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4541.0:1", 2020-03-11T22:55:00.849 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'119", 2020-03-11T22:55:00.849 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "119", 2020-03-11T22:55:00.849 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.849 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.849 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.849 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4544.0:1", 2020-03-11T22:55:00.850 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'120", 2020-03-11T22:55:00.850 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "120", 2020-03-11T22:55:00.850 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.850 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.850 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.850 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4547.0:1", 2020-03-11T22:55:00.850 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'121", 2020-03-11T22:55:00.851 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "121", 2020-03-11T22:55:00.851 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.851 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.851 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.851 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4550.0:1", 2020-03-11T22:55:00.851 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'122", 2020-03-11T22:55:00.851 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "122", 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4553.0:1", 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'123", 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "123", 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.852 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.853 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.853 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4556.0:1", 2020-03-11T22:55:00.853 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'124", 2020-03-11T22:55:00.853 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "124", 2020-03-11T22:55:00.853 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.853 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.853 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.854 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4559.0:1", 2020-03-11T22:55:00.854 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'125", 2020-03-11T22:55:00.854 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "125", 2020-03-11T22:55:00.854 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.854 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.854 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.855 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4562.0:1", 2020-03-11T22:55:00.855 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'126", 2020-03-11T22:55:00.855 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "126", 2020-03-11T22:55:00.855 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.855 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.855 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.855 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4565.0:1", 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'127", 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "127", 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4568.0:1", 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'128", 2020-03-11T22:55:00.856 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "128", 2020-03-11T22:55:00.857 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.857 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.857 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.857 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4571.0:1", 2020-03-11T22:55:00.857 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'129", 2020-03-11T22:55:00.857 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "129", 2020-03-11T22:55:00.857 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.858 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.858 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.858 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4574.0:1", 2020-03-11T22:55:00.858 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'130", 2020-03-11T22:55:00.858 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "130", 2020-03-11T22:55:00.858 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.858 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.859 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.859 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4577.0:1", 2020-03-11T22:55:00.859 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'131", 2020-03-11T22:55:00.859 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "131", 2020-03-11T22:55:00.859 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.859 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4580.0:1", 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'132", 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "132", 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.860 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4583.0:1", 2020-03-11T22:55:00.861 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'133", 2020-03-11T22:55:00.861 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "133", 2020-03-11T22:55:00.861 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.861 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.861 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.861 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4586.0:1", 2020-03-11T22:55:00.862 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'134", 2020-03-11T22:55:00.862 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "134", 2020-03-11T22:55:00.862 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.862 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.862 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.862 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4589.0:1", 2020-03-11T22:55:00.862 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'135", 2020-03-11T22:55:00.863 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "135", 2020-03-11T22:55:00.863 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.863 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.863 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.863 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4592.0:1", 2020-03-11T22:55:00.863 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'136", 2020-03-11T22:55:00.863 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "136", 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4595.0:1", 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'137", 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "137", 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.864 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.865 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.865 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4598.0:1", 2020-03-11T22:55:00.865 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'138", 2020-03-11T22:55:00.865 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "138", 2020-03-11T22:55:00.865 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.865 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.865 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.866 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4601.0:1", 2020-03-11T22:55:00.866 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'139", 2020-03-11T22:55:00.866 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "139", 2020-03-11T22:55:00.866 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.866 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.866 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.866 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4604.0:1", 2020-03-11T22:55:00.867 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'140", 2020-03-11T22:55:00.867 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "140", 2020-03-11T22:55:00.867 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.867 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.868 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.868 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4607.0:1", 2020-03-11T22:55:00.869 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'141", 2020-03-11T22:55:00.869 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "141", 2020-03-11T22:55:00.869 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.869 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.869 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.869 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4610.0:1", 2020-03-11T22:55:00.869 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'142", 2020-03-11T22:55:00.870 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "142", 2020-03-11T22:55:00.870 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.870 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.870 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.870 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4613.0:1", 2020-03-11T22:55:00.870 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'143", 2020-03-11T22:55:00.870 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "143", 2020-03-11T22:55:00.871 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.871 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.871 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.871 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4616.0:1", 2020-03-11T22:55:00.871 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'144", 2020-03-11T22:55:00.871 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "144", 2020-03-11T22:55:00.872 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.872 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.872 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.872 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4619.0:1", 2020-03-11T22:55:00.872 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'145", 2020-03-11T22:55:00.872 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "145", 2020-03-11T22:55:00.872 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.873 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.873 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.873 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4622.0:1", 2020-03-11T22:55:00.873 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'146", 2020-03-11T22:55:00.873 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "146", 2020-03-11T22:55:00.873 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:00.874 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:00.874 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:00.874 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4625.0:1", 2020-03-11T22:55:00.874 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'147", 2020-03-11T22:55:01.241 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "147", 2020-03-11T22:55:01.241 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:01.242 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:01.242 INFO:tasks.workunit.client.0.smithi130.stdout: { 2020-03-11T22:55:01.242 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4628.0:1", 2020-03-11T22:55:01.242 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "31'148", 2020-03-11T22:55:01.242 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "148", 2020-03-11T22:55:01.242 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2020-03-11T22:55:01.242 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:01.243 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2020-03-11T22:55:01.243 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2020-03-11T22:55:01.243 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_missing_t": { 2020-03-11T22:55:01.243 INFO:tasks.workunit.client.0.smithi130.stdout: "missing": [], 2020-03-11T22:55:01.243 INFO:tasks.workunit.client.0.smithi130.stdout: "may_include_deletes": true 2020-03-11T22:55:01.243 INFO:tasks.workunit.client.0.smithi130.stdout: } 2020-03-11T22:55:01.243 INFO:tasks.workunit.client.0.smithi130.stdout:} 2020-03-11T22:55:01.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:01.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:55:01.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:55:01.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:01.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=148 2020-03-11T22:55:01.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 148 '!=' 148 ']' 2020-03-11T22:55:01.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:55:01.252 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to max 2 2020-03-11T22:55:01.252 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to copy log version 31'150 2020-03-11T22:55:01.252 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to copy log version 31'149 2020-03-11T22:55:01.252 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:55:01.252 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 30'1 2020-03-11T22:55:01.253 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'2 2020-03-11T22:55:01.253 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'3 2020-03-11T22:55:01.253 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'4 2020-03-11T22:55:01.253 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'5 2020-03-11T22:55:01.253 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'6 2020-03-11T22:55:01.253 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'7 2020-03-11T22:55:01.253 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'8 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'9 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'10 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'11 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'12 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'13 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'14 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'15 2020-03-11T22:55:01.254 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'16 2020-03-11T22:55:01.255 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'17 2020-03-11T22:55:01.255 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'18 2020-03-11T22:55:01.255 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'19 2020-03-11T22:55:01.255 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'20 2020-03-11T22:55:01.255 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'21 2020-03-11T22:55:01.255 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'22 2020-03-11T22:55:01.255 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'23 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'24 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'25 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'26 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'27 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'28 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'29 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'30 2020-03-11T22:55:01.256 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'31 2020-03-11T22:55:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'32 2020-03-11T22:55:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'33 2020-03-11T22:55:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'34 2020-03-11T22:55:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'35 2020-03-11T22:55:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'36 2020-03-11T22:55:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'37 2020-03-11T22:55:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'38 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'39 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'40 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'41 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'42 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'43 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'44 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'45 2020-03-11T22:55:01.258 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'46 2020-03-11T22:55:01.259 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'47 2020-03-11T22:55:01.259 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'48 2020-03-11T22:55:01.259 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'49 2020-03-11T22:55:01.259 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'50 2020-03-11T22:55:01.259 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'51 2020-03-11T22:55:01.259 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'52 2020-03-11T22:55:01.259 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'53 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'54 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'55 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'56 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'57 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'58 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'59 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'60 2020-03-11T22:55:01.260 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'61 2020-03-11T22:55:01.261 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'62 2020-03-11T22:55:01.261 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'63 2020-03-11T22:55:01.261 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'64 2020-03-11T22:55:01.261 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'65 2020-03-11T22:55:01.261 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'66 2020-03-11T22:55:01.261 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'67 2020-03-11T22:55:01.261 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'68 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'69 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'70 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'71 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'72 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'73 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'74 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'75 2020-03-11T22:55:01.262 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'76 2020-03-11T22:55:01.263 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'77 2020-03-11T22:55:01.263 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'78 2020-03-11T22:55:01.263 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'79 2020-03-11T22:55:01.263 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'80 2020-03-11T22:55:01.263 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'81 2020-03-11T22:55:01.263 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'82 2020-03-11T22:55:01.263 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'83 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'84 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'85 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'86 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'87 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'88 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'89 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'90 2020-03-11T22:55:01.264 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'91 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'92 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'93 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'94 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'95 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'96 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'97 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'98 2020-03-11T22:55:01.265 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'99 2020-03-11T22:55:01.266 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'100 2020-03-11T22:55:01.266 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'101 2020-03-11T22:55:01.266 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'102 2020-03-11T22:55:01.266 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'103 2020-03-11T22:55:01.266 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'104 2020-03-11T22:55:01.266 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'105 2020-03-11T22:55:01.266 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'106 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'107 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'108 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'109 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'110 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'111 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'112 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'113 2020-03-11T22:55:01.267 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'114 2020-03-11T22:55:01.268 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'115 2020-03-11T22:55:01.268 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'116 2020-03-11T22:55:01.268 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'117 2020-03-11T22:55:01.268 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'118 2020-03-11T22:55:01.268 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'119 2020-03-11T22:55:01.268 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'120 2020-03-11T22:55:01.268 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'121 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'122 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'123 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'124 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'125 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'126 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'127 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'128 2020-03-11T22:55:01.269 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'129 2020-03-11T22:55:01.270 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'130 2020-03-11T22:55:01.270 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'131 2020-03-11T22:55:01.270 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'132 2020-03-11T22:55:01.270 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'133 2020-03-11T22:55:01.270 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'134 2020-03-11T22:55:01.270 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'135 2020-03-11T22:55:01.270 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'136 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'137 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'138 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'139 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'140 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:55:01.271 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:55:01.272 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:55:01.272 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:55:01.272 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:55:01.272 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:55:01.272 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to max 2 2020-03-11T22:55:01.272 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to copy log version 31'150 2020-03-11T22:55:01.272 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to copy log version 31'149 2020-03-11T22:55:01.273 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:55:01.273 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 30'1 2020-03-11T22:55:01.273 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'2 2020-03-11T22:55:01.273 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'3 2020-03-11T22:55:01.273 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'4 2020-03-11T22:55:01.273 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'5 2020-03-11T22:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'6 2020-03-11T22:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'7 2020-03-11T22:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'8 2020-03-11T22:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'9 2020-03-11T22:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'10 2020-03-11T22:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'11 2020-03-11T22:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'12 2020-03-11T22:55:01.275 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'13 2020-03-11T22:55:01.275 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'14 2020-03-11T22:55:01.275 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'15 2020-03-11T22:55:01.275 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'16 2020-03-11T22:55:01.275 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'17 2020-03-11T22:55:01.275 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'18 2020-03-11T22:55:01.276 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'19 2020-03-11T22:55:01.276 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'20 2020-03-11T22:55:01.276 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'21 2020-03-11T22:55:01.276 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'22 2020-03-11T22:55:01.276 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'23 2020-03-11T22:55:01.276 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'24 2020-03-11T22:55:01.277 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'25 2020-03-11T22:55:01.277 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'26 2020-03-11T22:55:01.277 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'27 2020-03-11T22:55:01.277 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'28 2020-03-11T22:55:01.277 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'29 2020-03-11T22:55:01.277 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'30 2020-03-11T22:55:01.277 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'31 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'32 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'33 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'34 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'35 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'36 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'37 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'38 2020-03-11T22:55:01.278 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'39 2020-03-11T22:55:01.279 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'40 2020-03-11T22:55:01.279 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'41 2020-03-11T22:55:01.279 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'42 2020-03-11T22:55:01.279 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'43 2020-03-11T22:55:01.279 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'44 2020-03-11T22:55:01.279 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'45 2020-03-11T22:55:01.279 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'46 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'47 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'48 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'49 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'50 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'51 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'52 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'53 2020-03-11T22:55:01.280 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'54 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'55 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'56 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'57 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'58 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'59 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'60 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'61 2020-03-11T22:55:01.281 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'62 2020-03-11T22:55:01.282 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'63 2020-03-11T22:55:01.282 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'64 2020-03-11T22:55:01.282 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'65 2020-03-11T22:55:01.282 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'66 2020-03-11T22:55:01.282 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.576+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'67 2020-03-11T22:55:01.282 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'68 2020-03-11T22:55:01.282 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'69 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'70 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'71 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'72 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'73 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'74 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'75 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'76 2020-03-11T22:55:01.283 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'77 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'78 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'79 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'80 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'81 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'82 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'83 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'84 2020-03-11T22:55:01.284 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'85 2020-03-11T22:55:01.285 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'86 2020-03-11T22:55:01.285 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'87 2020-03-11T22:55:01.285 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'88 2020-03-11T22:55:01.285 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'89 2020-03-11T22:55:01.285 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'90 2020-03-11T22:55:01.285 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'91 2020-03-11T22:55:01.285 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'92 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'93 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'94 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'95 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'96 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'97 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'98 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'99 2020-03-11T22:55:01.286 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'100 2020-03-11T22:55:01.287 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'101 2020-03-11T22:55:01.287 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'102 2020-03-11T22:55:01.287 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'103 2020-03-11T22:55:01.287 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'104 2020-03-11T22:55:01.287 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'105 2020-03-11T22:55:01.287 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'106 2020-03-11T22:55:01.287 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'107 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'108 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'109 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'110 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'111 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'112 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'113 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'114 2020-03-11T22:55:01.288 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'115 2020-03-11T22:55:01.289 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'116 2020-03-11T22:55:01.289 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'117 2020-03-11T22:55:01.289 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'118 2020-03-11T22:55:01.289 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'119 2020-03-11T22:55:01.289 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'120 2020-03-11T22:55:01.289 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'121 2020-03-11T22:55:01.289 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'122 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'123 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'124 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'125 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'126 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'127 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'128 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'129 2020-03-11T22:55:01.290 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'130 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'131 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'132 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'133 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'134 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'135 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'136 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'137 2020-03-11T22:55:01.291 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'138 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'139 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'140 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:55:01.292 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to max 2 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to copy log version 31'150 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to copy log version 31'149 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:55:01.293 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 30'1 2020-03-11T22:55:01.294 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'2 2020-03-11T22:55:01.294 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'3 2020-03-11T22:55:01.294 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'4 2020-03-11T22:55:01.294 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'5 2020-03-11T22:55:01.294 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'6 2020-03-11T22:55:01.294 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'7 2020-03-11T22:55:01.294 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'8 2020-03-11T22:55:01.295 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'9 2020-03-11T22:55:01.295 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'10 2020-03-11T22:55:01.295 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'11 2020-03-11T22:55:01.295 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'12 2020-03-11T22:55:01.295 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'13 2020-03-11T22:55:01.295 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'14 2020-03-11T22:55:01.295 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'15 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'16 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'17 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'18 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'19 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'20 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'21 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'22 2020-03-11T22:55:01.296 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'23 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'24 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'25 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'26 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'27 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'28 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'29 2020-03-11T22:55:01.297 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'30 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'31 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'32 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'33 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'34 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'35 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'36 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'37 2020-03-11T22:55:01.298 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'38 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'39 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'40 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'41 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'42 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'43 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'44 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'45 2020-03-11T22:55:01.299 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'46 2020-03-11T22:55:01.300 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'47 2020-03-11T22:55:01.300 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'48 2020-03-11T22:55:01.300 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'49 2020-03-11T22:55:01.300 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'50 2020-03-11T22:55:01.300 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'51 2020-03-11T22:55:01.300 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'52 2020-03-11T22:55:01.300 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'53 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'54 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'55 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'56 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'57 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'58 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'59 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'60 2020-03-11T22:55:01.301 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'61 2020-03-11T22:55:01.302 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'62 2020-03-11T22:55:01.302 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'63 2020-03-11T22:55:01.302 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'64 2020-03-11T22:55:01.302 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'65 2020-03-11T22:55:01.302 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'66 2020-03-11T22:55:01.302 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'67 2020-03-11T22:55:01.302 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'68 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'69 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'70 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'71 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'72 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'73 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'74 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'75 2020-03-11T22:55:01.303 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'76 2020-03-11T22:55:01.304 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'77 2020-03-11T22:55:01.304 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'78 2020-03-11T22:55:01.304 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'79 2020-03-11T22:55:01.304 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'80 2020-03-11T22:55:01.304 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'81 2020-03-11T22:55:01.304 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'82 2020-03-11T22:55:01.304 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'83 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'84 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'85 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'86 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'87 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'88 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'89 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'90 2020-03-11T22:55:01.305 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'91 2020-03-11T22:55:01.306 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'92 2020-03-11T22:55:01.306 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'93 2020-03-11T22:55:01.306 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'94 2020-03-11T22:55:01.306 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'95 2020-03-11T22:55:01.306 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'96 2020-03-11T22:55:01.306 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'97 2020-03-11T22:55:01.306 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'98 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'99 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'100 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'101 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'102 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'103 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'104 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'105 2020-03-11T22:55:01.307 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'106 2020-03-11T22:55:01.308 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'107 2020-03-11T22:55:01.308 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'108 2020-03-11T22:55:01.308 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'109 2020-03-11T22:55:01.308 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'110 2020-03-11T22:55:01.308 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'111 2020-03-11T22:55:01.308 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'112 2020-03-11T22:55:01.308 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'113 2020-03-11T22:55:01.309 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'114 2020-03-11T22:55:01.309 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'115 2020-03-11T22:55:01.309 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'116 2020-03-11T22:55:01.309 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'117 2020-03-11T22:55:01.309 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'118 2020-03-11T22:55:01.309 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'119 2020-03-11T22:55:01.309 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'120 2020-03-11T22:55:01.310 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'121 2020-03-11T22:55:01.310 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'122 2020-03-11T22:55:01.310 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'123 2020-03-11T22:55:01.310 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'124 2020-03-11T22:55:01.310 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'125 2020-03-11T22:55:01.310 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'126 2020-03-11T22:55:01.311 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'127 2020-03-11T22:55:01.311 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'128 2020-03-11T22:55:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'129 2020-03-11T22:55:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'130 2020-03-11T22:55:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'131 2020-03-11T22:55:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'132 2020-03-11T22:55:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'133 2020-03-11T22:55:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'134 2020-03-11T22:55:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'135 2020-03-11T22:55:01.318 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'136 2020-03-11T22:55:01.318 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'137 2020-03-11T22:55:01.318 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'138 2020-03-11T22:55:01.318 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'139 2020-03-11T22:55:01.318 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'140 2020-03-11T22:55:01.318 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:55:01.318 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:25.577+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_after v 31'150 2020-03-11T22:55:01.319 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:55:01.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 30'1 2020-03-11T22:55:01.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'2 2020-03-11T22:55:01.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'3 2020-03-11T22:55:01.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'4 2020-03-11T22:55:01.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'5 2020-03-11T22:55:01.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'6 2020-03-11T22:55:01.320 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'7 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'8 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'9 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'10 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'11 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'12 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'13 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'14 2020-03-11T22:55:01.321 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'15 2020-03-11T22:55:01.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'16 2020-03-11T22:55:01.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'17 2020-03-11T22:55:01.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'18 2020-03-11T22:55:01.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'19 2020-03-11T22:55:01.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'20 2020-03-11T22:55:01.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'21 2020-03-11T22:55:01.322 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'22 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'23 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'24 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'25 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'26 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'27 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'28 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'29 2020-03-11T22:55:01.323 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'30 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'31 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'32 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'33 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'34 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'35 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'36 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'37 2020-03-11T22:55:01.324 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'38 2020-03-11T22:55:01.325 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'39 2020-03-11T22:55:01.325 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'40 2020-03-11T22:55:01.325 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'41 2020-03-11T22:55:01.325 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'42 2020-03-11T22:55:01.325 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'43 2020-03-11T22:55:01.325 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'44 2020-03-11T22:55:01.325 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'45 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'46 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'47 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'48 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'49 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'50 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'51 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'52 2020-03-11T22:55:01.326 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'53 2020-03-11T22:55:01.327 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'54 2020-03-11T22:55:01.327 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'55 2020-03-11T22:55:01.327 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'56 2020-03-11T22:55:01.327 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'57 2020-03-11T22:55:01.327 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'58 2020-03-11T22:55:01.327 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'59 2020-03-11T22:55:01.327 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'60 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'61 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'62 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'63 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'64 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'65 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'66 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'67 2020-03-11T22:55:01.328 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'68 2020-03-11T22:55:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'69 2020-03-11T22:55:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'70 2020-03-11T22:55:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'71 2020-03-11T22:55:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'72 2020-03-11T22:55:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'73 2020-03-11T22:55:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'74 2020-03-11T22:55:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'75 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'76 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'77 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'78 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'79 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'80 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'81 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'82 2020-03-11T22:55:01.330 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'83 2020-03-11T22:55:01.331 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'84 2020-03-11T22:55:01.331 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'85 2020-03-11T22:55:01.331 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'86 2020-03-11T22:55:01.331 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'87 2020-03-11T22:55:01.331 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'88 2020-03-11T22:55:01.331 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'89 2020-03-11T22:55:01.331 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'90 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'91 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'92 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'93 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'94 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'95 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'96 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'97 2020-03-11T22:55:01.332 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'98 2020-03-11T22:55:01.333 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'99 2020-03-11T22:55:01.333 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup version 31'100 2020-03-11T22:55:01.333 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'101 2020-03-11T22:55:01.333 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'102 2020-03-11T22:55:01.333 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'103 2020-03-11T22:55:01.333 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'104 2020-03-11T22:55:01.333 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'105 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'106 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'107 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'108 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'109 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'110 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'111 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'112 2020-03-11T22:55:01.334 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'113 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'114 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'115 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'116 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'117 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'118 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'119 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'120 2020-03-11T22:55:01.335 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'121 2020-03-11T22:55:01.336 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'122 2020-03-11T22:55:01.336 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'123 2020-03-11T22:55:01.336 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'124 2020-03-11T22:55:01.336 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'125 2020-03-11T22:55:01.336 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'126 2020-03-11T22:55:01.336 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'127 2020-03-11T22:55:01.336 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'128 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'129 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'130 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'131 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'132 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'133 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'134 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'135 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'136 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'137 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'138 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'139 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'140 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:55:01.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:55:01.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:55:01.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:55:01.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:55:01.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:55:01.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'149 2020-03-11T22:55:01.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:53.094+0000 7f6bf007b700 20 copy_up_to/copy_after copy dup from log version 31'150 2020-03-11T22:55:01.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:01.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:55:01.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:55:01.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:01.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:55:01.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:01.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:01.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:01.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:01.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:01.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:01.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:01.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:01.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:01.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:01.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:01.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:01.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:01.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76578 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:55:01.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:01.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:55:01.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:01.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:01.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:01.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:01.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:01.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:01.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:01.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:01.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:01.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:01.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:01.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:55:01.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:01.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:01.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76578 2020-03-11T22:55:01.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:01.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:01.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:55:01.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:55:01.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:01.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76578 2020-03-11T22:55:01.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:55:01.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_1 td/osd-backfill-recovery-log 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:119: TEST_recovery_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:121: TEST_recovery_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 40 0 40 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:55:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=40 2020-03-11T22:55:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=0 2020-03-11T22:55:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=40 2020-03-11T22:55:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:55:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:55:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:55:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:55:01.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:55:01.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:55:01.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:55:01.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:55:01.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:01.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:01.600 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:01.600 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.600 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:01.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:01.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:55:01.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:55:01.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:55:01.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:01.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:01.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:01.769 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:01.770 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:01.770 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.770 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:01.770 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76578/ceph-mon.a.asok 2020-03-11T22:55:01.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:01.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76578/ceph-mon.a.asok config get fsid 2020-03-11T22:55:01.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:55:01.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:01.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:01.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:55:01.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:55:01.864 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:01.864 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:01.864 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:01.864 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:01.865 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:01.865 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:01.865 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76578/ceph-mon.a.asok 2020-03-11T22:55:01.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:01.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76578/ceph-mon.a.asok config get mon_host 2020-03-11T22:55:01.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:55:01.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:01.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:55:01.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:55:01.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:55:01.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:55:01.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:55:02.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:55:02.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:02.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:02.833 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:02.833 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:02.833 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:02.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:02.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:02.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:02.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:55:02.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:55:02.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:55:02.857 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:55:02.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:55:02.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:02.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:55:02.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:02.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:02.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:55:02.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:02.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:55:02.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:02.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:02.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:02.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:02.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:02.865 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:02.865 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:02.865 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:02.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:02.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:02.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:02.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:02.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:02.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:02.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:02.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:02.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:02.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:02.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:02.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:02.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:55:02.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:02.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e1293926-3960-4699-bd6b-7cb181b6f950 2020-03-11T22:55:02.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 e1293926-3960-4699-bd6b-7cb181b6f950' 2020-03-11T22:55:02.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:02.868 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 e1293926-3960-4699-bd6b-7cb181b6f950 2020-03-11T22:55:02.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBGbGleLewKNBAAuQjIHbIOa6QRLL742Xbe2A== 2020-03-11T22:55:02.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBGbGleLewKNBAAuQjIHbIOa6QRLL742Xbe2A=="}' 2020-03-11T22:55:02.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e1293926-3960-4699-bd6b-7cb181b6f950 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:03.223 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:55:03.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:03.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBGbGleLewKNBAAuQjIHbIOa6QRLL742Xbe2A== --osd-uuid e1293926-3960-4699-bd6b-7cb181b6f950 2020-03-11T22:55:03.258 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:03.254+0000 7faf953caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:03.260 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:03.256+0000 7faf953caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:03.261 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:03.257+0000 7faf953caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:03.261 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:03.257+0000 7faf953caec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:03.261 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:03.257+0000 7faf953caec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:03.262 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:03.257+0000 7faf953caec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:03.298 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:03.295+0000 7faf953caec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:55:06.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:55:06.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:06.260 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2020-03-11T22:55:06.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:55:06.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:06.720 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2020-03-11T22:55:06.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:55:06.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:06.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:06.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:06.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:06.743 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:06.739+0000 7f66743c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:06.744 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:06.741+0000 7f66743c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:06.746 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:06.743+0000 7f66743c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:07.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:55:07.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:55:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:07.070 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:55:07.265 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:07.259+0000 7f66743c7ec0 -1 Falling back to public interface 2020-03-11T22:55:07.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:08.006 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:08.002+0000 7f66743c7ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:55:08.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:08.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:08.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:08.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:08.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:08.429 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:55:08.795 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/93416,v1:127.0.0.1:6803/93416] [v2:127.0.0.1:6804/93416,v1:127.0.0.1:6805/93416] exists,up e1293926-3960-4699-bd6b-7cb181b6f950 2020-03-11T22:55:08.795 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 d2665839-8ea4-41da-a009-a5b65f48b97a 2020-03-11T22:55:08.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:08.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:08.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:08.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:08.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:55:08.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:08.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:08.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:55:08.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:08.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:55:08.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:08.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:08.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:08.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:08.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:08.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:08.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:08.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:55:08.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:08.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d2665839-8ea4-41da-a009-a5b65f48b97a 2020-03-11T22:55:08.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 d2665839-8ea4-41da-a009-a5b65f48b97a' 2020-03-11T22:55:08.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:08.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBMbGlepYgDMBAAH6b0SYmPo2FxplKOBs70og== 2020-03-11T22:55:08.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBMbGlepYgDMBAAH6b0SYmPo2FxplKOBs70og=="}' 2020-03-11T22:55:08.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d2665839-8ea4-41da-a009-a5b65f48b97a -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:55:09.329 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:55:09.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:55:09.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBMbGlepYgDMBAAH6b0SYmPo2FxplKOBs70og== --osd-uuid d2665839-8ea4-41da-a009-a5b65f48b97a 2020-03-11T22:55:09.363 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:09.359+0000 7f1e56ab8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:09.379 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:09.375+0000 7f1e56ab8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:09.381 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:09.378+0000 7f1e56ab8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:09.382 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:09.378+0000 7f1e56ab8ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:09.382 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:09.378+0000 7f1e56ab8ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:09.382 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:09.378+0000 7f1e56ab8ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:09.446 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:09.442+0000 7f1e56ab8ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:55:12.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:55:12.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:12.506 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2020-03-11T22:55:12.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:55:12.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:12.962 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2020-03-11T22:55:12.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:55:12.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:12.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:12.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:12.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:12.984 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:12.981+0000 7ff807d80ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.986 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:12.983+0000 7ff807d80ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.988 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:12.984+0000 7ff807d80ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:13.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:55:13.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:13.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:55:13.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:13.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:13.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:13.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:13.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:13.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:13.320 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:55:13.517 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:13.513+0000 7ff807d80ec0 -1 Falling back to public interface 2020-03-11T22:55:13.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:14.269 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:14.265+0000 7ff807d80ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:55:14.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:14.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:14.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:14.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:14.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:14.664 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:55:15.016 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/93779,v1:127.0.0.1:6811/93779] [v2:127.0.0.1:6812/93779,v1:127.0.0.1:6813/93779] exists,up d2665839-8ea4-41da-a009-a5b65f48b97a 2020-03-11T22:55:15.016 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 5b895cbe-5bda-4d6c-8ec4-cb1a02616e58 2020-03-11T22:55:15.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:15.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:15.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:15.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:15.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:55:15.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:55:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:55:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:55:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:55:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:55:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:15.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:15.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:15.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:15.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:15.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:15.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:15.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:15.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:15.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:55:15.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=5b895cbe-5bda-4d6c-8ec4-cb1a02616e58 2020-03-11T22:55:15.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 5b895cbe-5bda-4d6c-8ec4-cb1a02616e58' 2020-03-11T22:55:15.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:15.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBTbGletcOaARAA55zT2JF86b6GLzNH9ULXbw== 2020-03-11T22:55:15.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBTbGletcOaARAA55zT2JF86b6GLzNH9ULXbw=="}' 2020-03-11T22:55:15.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 5b895cbe-5bda-4d6c-8ec4-cb1a02616e58 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:55:15.514 INFO:tasks.workunit.client.0.smithi130.stdout:2 2020-03-11T22:55:15.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:55:15.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBTbGletcOaARAA55zT2JF86b6GLzNH9ULXbw== --osd-uuid 5b895cbe-5bda-4d6c-8ec4-cb1a02616e58 2020-03-11T22:55:15.549 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:15.545+0000 7fc75b53eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:15.589+0000 7fc75b53eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:15.590+0000 7fc75b53eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:15.591+0000 7fc75b53eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:15.591+0000 7fc75b53eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:15.591+0000 7fc75b53eec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:15.689+0000 7fc75b53eec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:55:18.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:55:18.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:18.750 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2020-03-11T22:55:18.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:55:18.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:19.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:55:19.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:19.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:19.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:19.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:19.201 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2020-03-11T22:55:19.222 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:19.219+0000 7f5160ecfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.224 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:19.220+0000 7f5160ecfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.225 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:19.221+0000 7f5160ecfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:55:19.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:19.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:55:19.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:19.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:19.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:19.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:19.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:19.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:55:19.546 INFO:tasks.workunit.client.0.smithi130.stdout:0 2020-03-11T22:55:19.749 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:19.745+0000 7f5160ecfec0 -1 Falling back to public interface 2020-03-11T22:55:19.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:20.500 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:20.496+0000 7f5160ecfec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:55:20.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:20.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:20.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:20.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:20.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:55:20.887 INFO:tasks.workunit.client.0.smithi130.stdout:1 2020-03-11T22:55:21.266 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/94134,v1:127.0.0.1:6819/94134] [v2:127.0.0.1:6820/94134,v1:127.0.0.1:6821/94134] exists,up 5b895cbe-5bda-4d6c-8ec4-cb1a02616e58 2020-03-11T22:55:21.266 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 44099dab-64db-42e3-baca-9320798611b3 2020-03-11T22:55:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:55:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:55:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:55:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:21.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:21.270 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:21.270 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:21.270 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76578 2020-03-11T22:55:21.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:21.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' 2020-03-11T22:55:21.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:21.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:21.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:21.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:21.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:21.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:21.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:21.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:21.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:21.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:21.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:55:21.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:21.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=44099dab-64db-42e3-baca-9320798611b3 2020-03-11T22:55:21.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 44099dab-64db-42e3-baca-9320798611b3' 2020-03-11T22:55:21.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:21.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBZbGleHXCDEBAA6C4Qe7e9k1PaSw2dRKO7Hw== 2020-03-11T22:55:21.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBZbGleHXCDEBAA6C4Qe7e9k1PaSw2dRKO7Hw=="}' 2020-03-11T22:55:21.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 44099dab-64db-42e3-baca-9320798611b3 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:55:21.807 INFO:tasks.workunit.client.0.smithi130.stdout:3 2020-03-11T22:55:21.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:55:21.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=ed051852-7d7f-42eb-8322-18c0a6aaa2d4 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76578/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZbGleHXCDEBAA6C4Qe7e9k1PaSw2dRKO7Hw== --osd-uuid 44099dab-64db-42e3-baca-9320798611b3 2020-03-11T22:55:21.842 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:21.839+0000 7ff65b7afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.868 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:21.862+0000 7ff65b7afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.868 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:21.863+0000 7ff65b7afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.868 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:21.864+0000 7ff65b7afec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:21.869 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:21.864+0000 7ff65b7afec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:21.869 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:21.864+0000 7ff65b7afec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:22.000 INFO:tasks.workunit.client.0.smithi130.stderr:2020-03-11T22:55:21.996+0000 7ff65b7afec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:55:25.034 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:25.035 DEBUG:teuthology.task.console_log:Killing console logger for smithi130 2020-03-11T22:55:25.035 DEBUG:teuthology.task.console_log:Killing console logger for smithi130 2020-03-11T22:55:25.035 DEBUG:teuthology.exit:Finished running handlers